curs testare - foundation

110
Testing Course explaining the ISTQB Cert if ied Test er Fou nd ation Level Syllabus 1

Upload: alexandru-ionut-pohontu

Post on 02-Jun-2018

243 views

Category:

Documents


0 download

TRANSCRIPT

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 1110

Testing Course

explaining the

ISTQB Certified Tester Foundation Level Syllabus

1

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 2110

111 Why is testing necessary ndash Why do we test(hellipthe common answer is ) To find bugs

but consider alsobullTo reduce the impact of the failures at the clientrsquos site (live defects) andensure that they will not affect costs amp profitabilitybullTo decrease the rate of failures (increase the productrsquos reliability)

bullTo improve the quality of the productbullTo ensure requirements are implemented fully amp correctlybullTo validate that the product is fit for its intended purposebullTo verify that required standards and legal requirements are metbullTo maintain the company reputation

Testing provides the productrsquos measure of quality

Can we test everything Exhaustive testing is possible

-No sorry helliptime amp resources make it impractical hellipbut instead-We must understand the risk to the clientrsquos business of the software notfunctioning correctlybullWe must manage and reduce risk carry out a Risk Analysis of the

applicationbullPrioritise tests to focus them (time amp resources) on the main areas of risk2

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 3110

111 Why is testing necessary - Testing goals and objectives

The main goals of testing

bull Find defectsbull Assess the level of quality of the software product and providing relatedinformation to the stakeholdersbull Prevent defectsbull Reduce risk of operational incidents

bull Increase the product qualityDifferent viewpoints and objectives

bull Unit amp integration test ndash find as many defects as possiblebull Acceptance testing ndash confirm that the system works as specified and that thequality is good enoughbull Testing metrics gathering ndash provide information to the project manager aboutthe product quality and the risks involved

bull Design tests early and review requirements ndash help prevent defects3

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 4110

112 Why is testing necessary - Testing Glossary

A programmer (or analyst) can make an error (mistake) which produces a defect(fault bug) in the programrsquos code If such a defect in the code is executed thesystem will fail to do what it should do (or it will do something it should not do)causing a failure

Error (mistake) = a human action that produces an incorrect resultDefect (bug) = a flaw that can cause the component or system to fail to perform its

required functionFailure = deviation of the component or system from its expected delivery serviceor result

Anomaly = any condition that deviates from expectations based on requirementsspecifications design documents user documentation standards or someonersquosperceptions or expectations

Defect masking = An occurrence in which one defect prevents the detection ofanother

4

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 5110

112 Why is testing necessary ndash Causes of the errorsDefects are caused by human errors

Why Because ofTime pressure - the more pressure we are under the more likely we are tomake mistakesbullCode complexity or new technology

bullToo many system interactionsbullRequirements not clearly defined changed amp not properly documentedbullWe make wrong assumptions about missing bits of informationbullPoor communicationbullPoor training

5

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 6110

112 Why is testing necessary - Causes of software defects -Defects taxonomy

6

(Boris Beizer)

bull Requirements (incorrect logic completeness verifiability documentation changes)bull Features and functionality (correctness missing case domain and boundarymessages exception mishandled)

bull Structural (control flow sequence data processing)bull Data (definition structure access handling)bull Implementation and Codingbull Integration (internal and external interfaces)bull System (architecture performance recovery partitioning environment)bull Test definition and execution (test design test execution documentation reporting)

(Cem Kaner (b1))bull User interface (functionality communication missing performance output)bull Error handling (prevention detection recovery)bull Boundary (numeric loops)

bull Calculation (wrong constants wrong operation order over amp underflow)bull Initialization (data item string loop control)bull Control flow (stop crash loop if-then-elsehellip)bull Data handling (data type parameter list values)bull Race amp Load conditions (event sequence no resources)bull Source and version control (old bug reappear)bull Testing (fail to notice fail to test fail to report)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 7110

113 Why is testing necessary - The role of testing in thesoftware life cycle

Testers do cooperate with Analysts ndash to review the specifications for completeness and correctnessensure that they are testablebull Designers ndash to improve interfaces testability and usabilitybull Programmers ndash to review the code and assess structural flawsbull Project manager ndash to estimate plan develop test cases perform tests andreport bugs to assess the quality and risksbull Quality assurance staff ndash to provide defect metricsbull Interactions with these project roles are very complex

RACI matrix (Responsible Accountable Consulted Informed)

7

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 8110

114 Why is testing necessary ndash What is quality

8

Quality (ISO) = The totality of the characteristics of an entity that bear on its

ability to satisfy stated or implied needs

bull there are many more definitions

Testing means not only to Verify (the thing is done right)hellip but also to Validate (the right thing is done)

Software quality includes reliability usability efficiency maintainability andportabilityRELIABILITY The ability of the software product to perform its requiredfunctions under stated conditions for a specified period of time or for aspecified number of operationsUSABILITY The capability of the software to be understood learned used

and attractive to the user when used under specified conditionsEFFICIENCY The capability of the software product to provide appropriateperformance relative to the amount of resources used under stated conditionsMAINTAINABILITY The ease with which a software product can be modifiedto correct defects modified to meet new requirements modified to make futuremaintenance easier or adapted to a changed environmentPORTABILITY The ease with which the software product can be transferredfrom one hardware or software environment to another

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 9110

114 Why is testing necessary - Testing and quality

Testing does not inject Quality

into the product but will measurethe level of the Quality of theproduct

Quality can be measured forbull progressbull variance (planned versus actual)

Measuring product quality bullFunctional compliance - functional software requirements testing

bullNon functional requirementsbullTest coverage criteriabullDefect count or defect trend criteria

9

1 1 4 Wh i i li ib

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 10110

114 Why is testing necessary ndash quality attributes

The QUINT model (extended ISO model)

10

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 11110

115 Why is testing necessary - How much testing is enough

when to stop testingThe five basic criteria often used to decide are

bull Previously defined coverage goals have been metbull The defect discovery rate has dropped below a previously defined thresholdbull The cost of finding the next defect exceeds the expected loss from that defectbull The project team reaches consensus that it is appropriate to release the productbull The manager decides to deliver the product

All these criteria are risk basedIt is important not depending on only one stopping criterion

Software Reliability Engineering can help also to determine when to stop testingby taking into consideration aspects like failure intensity

11

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 12110

12 What is testing - Defini tion of testing

12

Testing = the process concerned with planning the necessary static and dynamic

activities preparation and evaluation of software products and related deliverablesin order tobull determine that they satisfy specified requirementsbull demonstrate that they are fit for the intended usebull detect defects help and motivate the developers to fix thembull measure assess and improve the quality of the software product

Testing should be performed throughout the whole software life cycle

There are two basic types of testing execution and non-execution based

Other definitions

bull(IEEE) Testing = the process of analyzing a software item to detect the differencesbetween existing and required conditions and to evaluate its featuresbull(Myers (b3)) Testing = the process of executing a program with the intent of findingerrorsbull(Craig amp Jaskiel (b5)) Testing = a concurrent lifecycle process of engineering usingand maintaining test-ware in order to measure and improve the quality of thesoftware being tested

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 13110

12 What is testing ndash Testing ldquo schoolsrdquo

13

Analytic School - testing is rigorous academic and technicalbullTesting is a branch of CSMathematicsbullTesting techniques must have a logic-mathematical formbullKey Question Which techniques should we usebullRequire precise and detailed specifications

Factory School - testing is a way to measure progress with emphasis on cost and

repeatable standardsbullTesting must be managed amp cost effectivebullTesting validates the product amp measures development progressbullKey Questions How can we measure whether wersquore making progress When will we be donebullRequire clear boundaries between testing and other activities (startstop criteria)

bullEncourage standards (V-model) ldquobest practicesrdquo and certificationQuality School - emphasizes process amp quality acting as the gatekeeper bullSoftware quality requires disciplinebullTesters may need to police developers to follow the rulesbullKey Question Are we following a good process

bullTesting is a stepping stone to ldquoprocess improvementrdquoContext-Driven School - emphasizes people setting out to find the bugs that will bemost important to stakeholders

bullSoftware is created by people People set the contextbullTesting finds bugs acting as a skilled mental activitybullKey Question What testing would be most valuable right nowbullExpect changes Adapt testing plans based on test resultsbullTesting research requires empirical and psychological study

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 14110

13 General testing principles

14

bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects

bull Complete exhaustive testing is impossible good strategy and risk managementmust be used

bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs

bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)

bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped

bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)

bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 15110

13 General testing principles ndash heuristics of software testing

bullOperability - The better it works the more efficiently it can be tested

bullObservability - What we see is what we test

bull Controllability - The better we control the software the more the testing processcan be automated and optimized

bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing

bull Simplicity - The less there is to test the more quickly we can test it

bull Stability - The fewer the changes the fewer are the disruptions to testing

bull Understandability - The more information we will have the smarter we will test

bull Suitability - The more we know about the intended use of the software the

better we can organize our testing to find important bugs15

1 4 1 F d t l t t h

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 16110

141 Fundamental test process ndash phases

-Test Planning amp Test control

-Test Analysis amp Design

-Test Implementation ampExecution

-Evaluating exit criteria ampReporting

-Test Closure activities

16

1 4 1 Fundamental test process planning amp control

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 17110

141 Fundamental test process ndash planning amp control

Planning1 Determine scope

bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations

2 Determine risksbull Choose quality risk analysis method (eg FMEA)

bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule

bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies

4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria

Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions

17

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 18110

142 Fundamental test process ndash analysis amp design

bull Reviewing the test basis (such as requirements architecture designinterfaces)

bull Identifying test conditions or test requirements and required test data

based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-

conditionsbull Identify possible test Oracles

bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required

infrastructure and tools

(see Lee Copeland (b2))

18

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 19110

142 Fundamental test process ndash what is a test Oracle

19

bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be

executed The Test Oracle

Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative

hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead

lsquoproblem - no problemrsquo

Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles

Possible issues

bull false alarmsbull missed bugs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 20110

143 Fundamental test process ndash implementation amp execution

Test implementation

bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)

bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission

(see Rex Black (b4))

20

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 21110

143 Fundamental test process ndash priori tizing the Test Cases

Why prioritize the Test Cases

bullIt is not possible to test everything we must do our best in the time available

bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence

bullThis means we must prioritise and focus testing on the priorities

What to watch

bullSeverity of possible defectsbullProbability of possible defects

bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity

21

144 Fundamental test process ndash evaluating exit criteria and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 22110

144 Fundamental test process evaluating exit criteria andreporting

Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed

bull Check if testing mission should be changed

Test report ing

bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity

(see Rex Black (b4) amp RUP- Test discipline(s5))

22

1 4 5 F d l l

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 23110

145 Fundamental test process ndash test closure

bull Verify if test deliverables have been delivered

bull Check and close the remaining active bug reports

bull Archiving the test-ware and environment

bull Handover of the test environment

bull Analyze the identified test process problems (lessons learned)

bull Implement action plan based improvements

(see Rex Black (b4))

23

1 5 Th h l f i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 24110

15 The psychology of testing

24

Testing is regarded as a lsquodestructiversquo activity

(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts

bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise

bullShould be a planned organised kind ofperson

Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices

Rex BlackrsquosTop 10 professional errors

bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations

(see also Brian Marickrsquos article )

1 5 Th h l g f t ti g

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 25110

15 The psychology of testing

25

ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)

ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects

bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs

Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)

Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts

bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence

2 1 1 The V testing model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 26110

211 The V testing model

26

211 The V testing model ndash Verification amp Validationifi i C fi i b

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 2110

111 Why is testing necessary ndash Why do we test(hellipthe common answer is ) To find bugs

but consider alsobullTo reduce the impact of the failures at the clientrsquos site (live defects) andensure that they will not affect costs amp profitabilitybullTo decrease the rate of failures (increase the productrsquos reliability)

bullTo improve the quality of the productbullTo ensure requirements are implemented fully amp correctlybullTo validate that the product is fit for its intended purposebullTo verify that required standards and legal requirements are metbullTo maintain the company reputation

Testing provides the productrsquos measure of quality

Can we test everything Exhaustive testing is possible

-No sorry helliptime amp resources make it impractical hellipbut instead-We must understand the risk to the clientrsquos business of the software notfunctioning correctlybullWe must manage and reduce risk carry out a Risk Analysis of the

applicationbullPrioritise tests to focus them (time amp resources) on the main areas of risk2

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 3110

111 Why is testing necessary - Testing goals and objectives

The main goals of testing

bull Find defectsbull Assess the level of quality of the software product and providing relatedinformation to the stakeholdersbull Prevent defectsbull Reduce risk of operational incidents

bull Increase the product qualityDifferent viewpoints and objectives

bull Unit amp integration test ndash find as many defects as possiblebull Acceptance testing ndash confirm that the system works as specified and that thequality is good enoughbull Testing metrics gathering ndash provide information to the project manager aboutthe product quality and the risks involved

bull Design tests early and review requirements ndash help prevent defects3

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 4110

112 Why is testing necessary - Testing Glossary

A programmer (or analyst) can make an error (mistake) which produces a defect(fault bug) in the programrsquos code If such a defect in the code is executed thesystem will fail to do what it should do (or it will do something it should not do)causing a failure

Error (mistake) = a human action that produces an incorrect resultDefect (bug) = a flaw that can cause the component or system to fail to perform its

required functionFailure = deviation of the component or system from its expected delivery serviceor result

Anomaly = any condition that deviates from expectations based on requirementsspecifications design documents user documentation standards or someonersquosperceptions or expectations

Defect masking = An occurrence in which one defect prevents the detection ofanother

4

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 5110

112 Why is testing necessary ndash Causes of the errorsDefects are caused by human errors

Why Because ofTime pressure - the more pressure we are under the more likely we are tomake mistakesbullCode complexity or new technology

bullToo many system interactionsbullRequirements not clearly defined changed amp not properly documentedbullWe make wrong assumptions about missing bits of informationbullPoor communicationbullPoor training

5

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 6110

112 Why is testing necessary - Causes of software defects -Defects taxonomy

6

(Boris Beizer)

bull Requirements (incorrect logic completeness verifiability documentation changes)bull Features and functionality (correctness missing case domain and boundarymessages exception mishandled)

bull Structural (control flow sequence data processing)bull Data (definition structure access handling)bull Implementation and Codingbull Integration (internal and external interfaces)bull System (architecture performance recovery partitioning environment)bull Test definition and execution (test design test execution documentation reporting)

(Cem Kaner (b1))bull User interface (functionality communication missing performance output)bull Error handling (prevention detection recovery)bull Boundary (numeric loops)

bull Calculation (wrong constants wrong operation order over amp underflow)bull Initialization (data item string loop control)bull Control flow (stop crash loop if-then-elsehellip)bull Data handling (data type parameter list values)bull Race amp Load conditions (event sequence no resources)bull Source and version control (old bug reappear)bull Testing (fail to notice fail to test fail to report)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 7110

113 Why is testing necessary - The role of testing in thesoftware life cycle

Testers do cooperate with Analysts ndash to review the specifications for completeness and correctnessensure that they are testablebull Designers ndash to improve interfaces testability and usabilitybull Programmers ndash to review the code and assess structural flawsbull Project manager ndash to estimate plan develop test cases perform tests andreport bugs to assess the quality and risksbull Quality assurance staff ndash to provide defect metricsbull Interactions with these project roles are very complex

RACI matrix (Responsible Accountable Consulted Informed)

7

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 8110

114 Why is testing necessary ndash What is quality

8

Quality (ISO) = The totality of the characteristics of an entity that bear on its

ability to satisfy stated or implied needs

bull there are many more definitions

Testing means not only to Verify (the thing is done right)hellip but also to Validate (the right thing is done)

Software quality includes reliability usability efficiency maintainability andportabilityRELIABILITY The ability of the software product to perform its requiredfunctions under stated conditions for a specified period of time or for aspecified number of operationsUSABILITY The capability of the software to be understood learned used

and attractive to the user when used under specified conditionsEFFICIENCY The capability of the software product to provide appropriateperformance relative to the amount of resources used under stated conditionsMAINTAINABILITY The ease with which a software product can be modifiedto correct defects modified to meet new requirements modified to make futuremaintenance easier or adapted to a changed environmentPORTABILITY The ease with which the software product can be transferredfrom one hardware or software environment to another

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 9110

114 Why is testing necessary - Testing and quality

Testing does not inject Quality

into the product but will measurethe level of the Quality of theproduct

Quality can be measured forbull progressbull variance (planned versus actual)

Measuring product quality bullFunctional compliance - functional software requirements testing

bullNon functional requirementsbullTest coverage criteriabullDefect count or defect trend criteria

9

1 1 4 Wh i i li ib

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 10110

114 Why is testing necessary ndash quality attributes

The QUINT model (extended ISO model)

10

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 11110

115 Why is testing necessary - How much testing is enough

when to stop testingThe five basic criteria often used to decide are

bull Previously defined coverage goals have been metbull The defect discovery rate has dropped below a previously defined thresholdbull The cost of finding the next defect exceeds the expected loss from that defectbull The project team reaches consensus that it is appropriate to release the productbull The manager decides to deliver the product

All these criteria are risk basedIt is important not depending on only one stopping criterion

Software Reliability Engineering can help also to determine when to stop testingby taking into consideration aspects like failure intensity

11

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 12110

12 What is testing - Defini tion of testing

12

Testing = the process concerned with planning the necessary static and dynamic

activities preparation and evaluation of software products and related deliverablesin order tobull determine that they satisfy specified requirementsbull demonstrate that they are fit for the intended usebull detect defects help and motivate the developers to fix thembull measure assess and improve the quality of the software product

Testing should be performed throughout the whole software life cycle

There are two basic types of testing execution and non-execution based

Other definitions

bull(IEEE) Testing = the process of analyzing a software item to detect the differencesbetween existing and required conditions and to evaluate its featuresbull(Myers (b3)) Testing = the process of executing a program with the intent of findingerrorsbull(Craig amp Jaskiel (b5)) Testing = a concurrent lifecycle process of engineering usingand maintaining test-ware in order to measure and improve the quality of thesoftware being tested

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 13110

12 What is testing ndash Testing ldquo schoolsrdquo

13

Analytic School - testing is rigorous academic and technicalbullTesting is a branch of CSMathematicsbullTesting techniques must have a logic-mathematical formbullKey Question Which techniques should we usebullRequire precise and detailed specifications

Factory School - testing is a way to measure progress with emphasis on cost and

repeatable standardsbullTesting must be managed amp cost effectivebullTesting validates the product amp measures development progressbullKey Questions How can we measure whether wersquore making progress When will we be donebullRequire clear boundaries between testing and other activities (startstop criteria)

bullEncourage standards (V-model) ldquobest practicesrdquo and certificationQuality School - emphasizes process amp quality acting as the gatekeeper bullSoftware quality requires disciplinebullTesters may need to police developers to follow the rulesbullKey Question Are we following a good process

bullTesting is a stepping stone to ldquoprocess improvementrdquoContext-Driven School - emphasizes people setting out to find the bugs that will bemost important to stakeholders

bullSoftware is created by people People set the contextbullTesting finds bugs acting as a skilled mental activitybullKey Question What testing would be most valuable right nowbullExpect changes Adapt testing plans based on test resultsbullTesting research requires empirical and psychological study

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 14110

13 General testing principles

14

bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects

bull Complete exhaustive testing is impossible good strategy and risk managementmust be used

bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs

bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)

bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped

bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)

bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 15110

13 General testing principles ndash heuristics of software testing

bullOperability - The better it works the more efficiently it can be tested

bullObservability - What we see is what we test

bull Controllability - The better we control the software the more the testing processcan be automated and optimized

bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing

bull Simplicity - The less there is to test the more quickly we can test it

bull Stability - The fewer the changes the fewer are the disruptions to testing

bull Understandability - The more information we will have the smarter we will test

bull Suitability - The more we know about the intended use of the software the

better we can organize our testing to find important bugs15

1 4 1 F d t l t t h

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 16110

141 Fundamental test process ndash phases

-Test Planning amp Test control

-Test Analysis amp Design

-Test Implementation ampExecution

-Evaluating exit criteria ampReporting

-Test Closure activities

16

1 4 1 Fundamental test process planning amp control

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 17110

141 Fundamental test process ndash planning amp control

Planning1 Determine scope

bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations

2 Determine risksbull Choose quality risk analysis method (eg FMEA)

bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule

bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies

4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria

Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions

17

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 18110

142 Fundamental test process ndash analysis amp design

bull Reviewing the test basis (such as requirements architecture designinterfaces)

bull Identifying test conditions or test requirements and required test data

based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-

conditionsbull Identify possible test Oracles

bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required

infrastructure and tools

(see Lee Copeland (b2))

18

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 19110

142 Fundamental test process ndash what is a test Oracle

19

bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be

executed The Test Oracle

Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative

hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead

lsquoproblem - no problemrsquo

Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles

Possible issues

bull false alarmsbull missed bugs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 20110

143 Fundamental test process ndash implementation amp execution

Test implementation

bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)

bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission

(see Rex Black (b4))

20

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 21110

143 Fundamental test process ndash priori tizing the Test Cases

Why prioritize the Test Cases

bullIt is not possible to test everything we must do our best in the time available

bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence

bullThis means we must prioritise and focus testing on the priorities

What to watch

bullSeverity of possible defectsbullProbability of possible defects

bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity

21

144 Fundamental test process ndash evaluating exit criteria and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 22110

144 Fundamental test process evaluating exit criteria andreporting

Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed

bull Check if testing mission should be changed

Test report ing

bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity

(see Rex Black (b4) amp RUP- Test discipline(s5))

22

1 4 5 F d l l

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 23110

145 Fundamental test process ndash test closure

bull Verify if test deliverables have been delivered

bull Check and close the remaining active bug reports

bull Archiving the test-ware and environment

bull Handover of the test environment

bull Analyze the identified test process problems (lessons learned)

bull Implement action plan based improvements

(see Rex Black (b4))

23

1 5 Th h l f i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 24110

15 The psychology of testing

24

Testing is regarded as a lsquodestructiversquo activity

(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts

bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise

bullShould be a planned organised kind ofperson

Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices

Rex BlackrsquosTop 10 professional errors

bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations

(see also Brian Marickrsquos article )

1 5 Th h l g f t ti g

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 25110

15 The psychology of testing

25

ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)

ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects

bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs

Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)

Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts

bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence

2 1 1 The V testing model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 26110

211 The V testing model

26

211 The V testing model ndash Verification amp Validationifi i C fi i b

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 3110

111 Why is testing necessary - Testing goals and objectives

The main goals of testing

bull Find defectsbull Assess the level of quality of the software product and providing relatedinformation to the stakeholdersbull Prevent defectsbull Reduce risk of operational incidents

bull Increase the product qualityDifferent viewpoints and objectives

bull Unit amp integration test ndash find as many defects as possiblebull Acceptance testing ndash confirm that the system works as specified and that thequality is good enoughbull Testing metrics gathering ndash provide information to the project manager aboutthe product quality and the risks involved

bull Design tests early and review requirements ndash help prevent defects3

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 4110

112 Why is testing necessary - Testing Glossary

A programmer (or analyst) can make an error (mistake) which produces a defect(fault bug) in the programrsquos code If such a defect in the code is executed thesystem will fail to do what it should do (or it will do something it should not do)causing a failure

Error (mistake) = a human action that produces an incorrect resultDefect (bug) = a flaw that can cause the component or system to fail to perform its

required functionFailure = deviation of the component or system from its expected delivery serviceor result

Anomaly = any condition that deviates from expectations based on requirementsspecifications design documents user documentation standards or someonersquosperceptions or expectations

Defect masking = An occurrence in which one defect prevents the detection ofanother

4

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 5110

112 Why is testing necessary ndash Causes of the errorsDefects are caused by human errors

Why Because ofTime pressure - the more pressure we are under the more likely we are tomake mistakesbullCode complexity or new technology

bullToo many system interactionsbullRequirements not clearly defined changed amp not properly documentedbullWe make wrong assumptions about missing bits of informationbullPoor communicationbullPoor training

5

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 6110

112 Why is testing necessary - Causes of software defects -Defects taxonomy

6

(Boris Beizer)

bull Requirements (incorrect logic completeness verifiability documentation changes)bull Features and functionality (correctness missing case domain and boundarymessages exception mishandled)

bull Structural (control flow sequence data processing)bull Data (definition structure access handling)bull Implementation and Codingbull Integration (internal and external interfaces)bull System (architecture performance recovery partitioning environment)bull Test definition and execution (test design test execution documentation reporting)

(Cem Kaner (b1))bull User interface (functionality communication missing performance output)bull Error handling (prevention detection recovery)bull Boundary (numeric loops)

bull Calculation (wrong constants wrong operation order over amp underflow)bull Initialization (data item string loop control)bull Control flow (stop crash loop if-then-elsehellip)bull Data handling (data type parameter list values)bull Race amp Load conditions (event sequence no resources)bull Source and version control (old bug reappear)bull Testing (fail to notice fail to test fail to report)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 7110

113 Why is testing necessary - The role of testing in thesoftware life cycle

Testers do cooperate with Analysts ndash to review the specifications for completeness and correctnessensure that they are testablebull Designers ndash to improve interfaces testability and usabilitybull Programmers ndash to review the code and assess structural flawsbull Project manager ndash to estimate plan develop test cases perform tests andreport bugs to assess the quality and risksbull Quality assurance staff ndash to provide defect metricsbull Interactions with these project roles are very complex

RACI matrix (Responsible Accountable Consulted Informed)

7

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 8110

114 Why is testing necessary ndash What is quality

8

Quality (ISO) = The totality of the characteristics of an entity that bear on its

ability to satisfy stated or implied needs

bull there are many more definitions

Testing means not only to Verify (the thing is done right)hellip but also to Validate (the right thing is done)

Software quality includes reliability usability efficiency maintainability andportabilityRELIABILITY The ability of the software product to perform its requiredfunctions under stated conditions for a specified period of time or for aspecified number of operationsUSABILITY The capability of the software to be understood learned used

and attractive to the user when used under specified conditionsEFFICIENCY The capability of the software product to provide appropriateperformance relative to the amount of resources used under stated conditionsMAINTAINABILITY The ease with which a software product can be modifiedto correct defects modified to meet new requirements modified to make futuremaintenance easier or adapted to a changed environmentPORTABILITY The ease with which the software product can be transferredfrom one hardware or software environment to another

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 9110

114 Why is testing necessary - Testing and quality

Testing does not inject Quality

into the product but will measurethe level of the Quality of theproduct

Quality can be measured forbull progressbull variance (planned versus actual)

Measuring product quality bullFunctional compliance - functional software requirements testing

bullNon functional requirementsbullTest coverage criteriabullDefect count or defect trend criteria

9

1 1 4 Wh i i li ib

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 10110

114 Why is testing necessary ndash quality attributes

The QUINT model (extended ISO model)

10

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 11110

115 Why is testing necessary - How much testing is enough

when to stop testingThe five basic criteria often used to decide are

bull Previously defined coverage goals have been metbull The defect discovery rate has dropped below a previously defined thresholdbull The cost of finding the next defect exceeds the expected loss from that defectbull The project team reaches consensus that it is appropriate to release the productbull The manager decides to deliver the product

All these criteria are risk basedIt is important not depending on only one stopping criterion

Software Reliability Engineering can help also to determine when to stop testingby taking into consideration aspects like failure intensity

11

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 12110

12 What is testing - Defini tion of testing

12

Testing = the process concerned with planning the necessary static and dynamic

activities preparation and evaluation of software products and related deliverablesin order tobull determine that they satisfy specified requirementsbull demonstrate that they are fit for the intended usebull detect defects help and motivate the developers to fix thembull measure assess and improve the quality of the software product

Testing should be performed throughout the whole software life cycle

There are two basic types of testing execution and non-execution based

Other definitions

bull(IEEE) Testing = the process of analyzing a software item to detect the differencesbetween existing and required conditions and to evaluate its featuresbull(Myers (b3)) Testing = the process of executing a program with the intent of findingerrorsbull(Craig amp Jaskiel (b5)) Testing = a concurrent lifecycle process of engineering usingand maintaining test-ware in order to measure and improve the quality of thesoftware being tested

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 13110

12 What is testing ndash Testing ldquo schoolsrdquo

13

Analytic School - testing is rigorous academic and technicalbullTesting is a branch of CSMathematicsbullTesting techniques must have a logic-mathematical formbullKey Question Which techniques should we usebullRequire precise and detailed specifications

Factory School - testing is a way to measure progress with emphasis on cost and

repeatable standardsbullTesting must be managed amp cost effectivebullTesting validates the product amp measures development progressbullKey Questions How can we measure whether wersquore making progress When will we be donebullRequire clear boundaries between testing and other activities (startstop criteria)

bullEncourage standards (V-model) ldquobest practicesrdquo and certificationQuality School - emphasizes process amp quality acting as the gatekeeper bullSoftware quality requires disciplinebullTesters may need to police developers to follow the rulesbullKey Question Are we following a good process

bullTesting is a stepping stone to ldquoprocess improvementrdquoContext-Driven School - emphasizes people setting out to find the bugs that will bemost important to stakeholders

bullSoftware is created by people People set the contextbullTesting finds bugs acting as a skilled mental activitybullKey Question What testing would be most valuable right nowbullExpect changes Adapt testing plans based on test resultsbullTesting research requires empirical and psychological study

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 14110

13 General testing principles

14

bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects

bull Complete exhaustive testing is impossible good strategy and risk managementmust be used

bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs

bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)

bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped

bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)

bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 15110

13 General testing principles ndash heuristics of software testing

bullOperability - The better it works the more efficiently it can be tested

bullObservability - What we see is what we test

bull Controllability - The better we control the software the more the testing processcan be automated and optimized

bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing

bull Simplicity - The less there is to test the more quickly we can test it

bull Stability - The fewer the changes the fewer are the disruptions to testing

bull Understandability - The more information we will have the smarter we will test

bull Suitability - The more we know about the intended use of the software the

better we can organize our testing to find important bugs15

1 4 1 F d t l t t h

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 16110

141 Fundamental test process ndash phases

-Test Planning amp Test control

-Test Analysis amp Design

-Test Implementation ampExecution

-Evaluating exit criteria ampReporting

-Test Closure activities

16

1 4 1 Fundamental test process planning amp control

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 17110

141 Fundamental test process ndash planning amp control

Planning1 Determine scope

bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations

2 Determine risksbull Choose quality risk analysis method (eg FMEA)

bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule

bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies

4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria

Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions

17

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 18110

142 Fundamental test process ndash analysis amp design

bull Reviewing the test basis (such as requirements architecture designinterfaces)

bull Identifying test conditions or test requirements and required test data

based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-

conditionsbull Identify possible test Oracles

bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required

infrastructure and tools

(see Lee Copeland (b2))

18

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 19110

142 Fundamental test process ndash what is a test Oracle

19

bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be

executed The Test Oracle

Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative

hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead

lsquoproblem - no problemrsquo

Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles

Possible issues

bull false alarmsbull missed bugs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 20110

143 Fundamental test process ndash implementation amp execution

Test implementation

bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)

bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission

(see Rex Black (b4))

20

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 21110

143 Fundamental test process ndash priori tizing the Test Cases

Why prioritize the Test Cases

bullIt is not possible to test everything we must do our best in the time available

bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence

bullThis means we must prioritise and focus testing on the priorities

What to watch

bullSeverity of possible defectsbullProbability of possible defects

bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity

21

144 Fundamental test process ndash evaluating exit criteria and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 22110

144 Fundamental test process evaluating exit criteria andreporting

Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed

bull Check if testing mission should be changed

Test report ing

bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity

(see Rex Black (b4) amp RUP- Test discipline(s5))

22

1 4 5 F d l l

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 23110

145 Fundamental test process ndash test closure

bull Verify if test deliverables have been delivered

bull Check and close the remaining active bug reports

bull Archiving the test-ware and environment

bull Handover of the test environment

bull Analyze the identified test process problems (lessons learned)

bull Implement action plan based improvements

(see Rex Black (b4))

23

1 5 Th h l f i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 24110

15 The psychology of testing

24

Testing is regarded as a lsquodestructiversquo activity

(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts

bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise

bullShould be a planned organised kind ofperson

Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices

Rex BlackrsquosTop 10 professional errors

bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations

(see also Brian Marickrsquos article )

1 5 Th h l g f t ti g

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 25110

15 The psychology of testing

25

ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)

ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects

bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs

Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)

Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts

bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence

2 1 1 The V testing model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 26110

211 The V testing model

26

211 The V testing model ndash Verification amp Validationifi i C fi i b

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 4110

112 Why is testing necessary - Testing Glossary

A programmer (or analyst) can make an error (mistake) which produces a defect(fault bug) in the programrsquos code If such a defect in the code is executed thesystem will fail to do what it should do (or it will do something it should not do)causing a failure

Error (mistake) = a human action that produces an incorrect resultDefect (bug) = a flaw that can cause the component or system to fail to perform its

required functionFailure = deviation of the component or system from its expected delivery serviceor result

Anomaly = any condition that deviates from expectations based on requirementsspecifications design documents user documentation standards or someonersquosperceptions or expectations

Defect masking = An occurrence in which one defect prevents the detection ofanother

4

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 5110

112 Why is testing necessary ndash Causes of the errorsDefects are caused by human errors

Why Because ofTime pressure - the more pressure we are under the more likely we are tomake mistakesbullCode complexity or new technology

bullToo many system interactionsbullRequirements not clearly defined changed amp not properly documentedbullWe make wrong assumptions about missing bits of informationbullPoor communicationbullPoor training

5

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 6110

112 Why is testing necessary - Causes of software defects -Defects taxonomy

6

(Boris Beizer)

bull Requirements (incorrect logic completeness verifiability documentation changes)bull Features and functionality (correctness missing case domain and boundarymessages exception mishandled)

bull Structural (control flow sequence data processing)bull Data (definition structure access handling)bull Implementation and Codingbull Integration (internal and external interfaces)bull System (architecture performance recovery partitioning environment)bull Test definition and execution (test design test execution documentation reporting)

(Cem Kaner (b1))bull User interface (functionality communication missing performance output)bull Error handling (prevention detection recovery)bull Boundary (numeric loops)

bull Calculation (wrong constants wrong operation order over amp underflow)bull Initialization (data item string loop control)bull Control flow (stop crash loop if-then-elsehellip)bull Data handling (data type parameter list values)bull Race amp Load conditions (event sequence no resources)bull Source and version control (old bug reappear)bull Testing (fail to notice fail to test fail to report)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 7110

113 Why is testing necessary - The role of testing in thesoftware life cycle

Testers do cooperate with Analysts ndash to review the specifications for completeness and correctnessensure that they are testablebull Designers ndash to improve interfaces testability and usabilitybull Programmers ndash to review the code and assess structural flawsbull Project manager ndash to estimate plan develop test cases perform tests andreport bugs to assess the quality and risksbull Quality assurance staff ndash to provide defect metricsbull Interactions with these project roles are very complex

RACI matrix (Responsible Accountable Consulted Informed)

7

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 8110

114 Why is testing necessary ndash What is quality

8

Quality (ISO) = The totality of the characteristics of an entity that bear on its

ability to satisfy stated or implied needs

bull there are many more definitions

Testing means not only to Verify (the thing is done right)hellip but also to Validate (the right thing is done)

Software quality includes reliability usability efficiency maintainability andportabilityRELIABILITY The ability of the software product to perform its requiredfunctions under stated conditions for a specified period of time or for aspecified number of operationsUSABILITY The capability of the software to be understood learned used

and attractive to the user when used under specified conditionsEFFICIENCY The capability of the software product to provide appropriateperformance relative to the amount of resources used under stated conditionsMAINTAINABILITY The ease with which a software product can be modifiedto correct defects modified to meet new requirements modified to make futuremaintenance easier or adapted to a changed environmentPORTABILITY The ease with which the software product can be transferredfrom one hardware or software environment to another

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 9110

114 Why is testing necessary - Testing and quality

Testing does not inject Quality

into the product but will measurethe level of the Quality of theproduct

Quality can be measured forbull progressbull variance (planned versus actual)

Measuring product quality bullFunctional compliance - functional software requirements testing

bullNon functional requirementsbullTest coverage criteriabullDefect count or defect trend criteria

9

1 1 4 Wh i i li ib

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 10110

114 Why is testing necessary ndash quality attributes

The QUINT model (extended ISO model)

10

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 11110

115 Why is testing necessary - How much testing is enough

when to stop testingThe five basic criteria often used to decide are

bull Previously defined coverage goals have been metbull The defect discovery rate has dropped below a previously defined thresholdbull The cost of finding the next defect exceeds the expected loss from that defectbull The project team reaches consensus that it is appropriate to release the productbull The manager decides to deliver the product

All these criteria are risk basedIt is important not depending on only one stopping criterion

Software Reliability Engineering can help also to determine when to stop testingby taking into consideration aspects like failure intensity

11

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 12110

12 What is testing - Defini tion of testing

12

Testing = the process concerned with planning the necessary static and dynamic

activities preparation and evaluation of software products and related deliverablesin order tobull determine that they satisfy specified requirementsbull demonstrate that they are fit for the intended usebull detect defects help and motivate the developers to fix thembull measure assess and improve the quality of the software product

Testing should be performed throughout the whole software life cycle

There are two basic types of testing execution and non-execution based

Other definitions

bull(IEEE) Testing = the process of analyzing a software item to detect the differencesbetween existing and required conditions and to evaluate its featuresbull(Myers (b3)) Testing = the process of executing a program with the intent of findingerrorsbull(Craig amp Jaskiel (b5)) Testing = a concurrent lifecycle process of engineering usingand maintaining test-ware in order to measure and improve the quality of thesoftware being tested

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 13110

12 What is testing ndash Testing ldquo schoolsrdquo

13

Analytic School - testing is rigorous academic and technicalbullTesting is a branch of CSMathematicsbullTesting techniques must have a logic-mathematical formbullKey Question Which techniques should we usebullRequire precise and detailed specifications

Factory School - testing is a way to measure progress with emphasis on cost and

repeatable standardsbullTesting must be managed amp cost effectivebullTesting validates the product amp measures development progressbullKey Questions How can we measure whether wersquore making progress When will we be donebullRequire clear boundaries between testing and other activities (startstop criteria)

bullEncourage standards (V-model) ldquobest practicesrdquo and certificationQuality School - emphasizes process amp quality acting as the gatekeeper bullSoftware quality requires disciplinebullTesters may need to police developers to follow the rulesbullKey Question Are we following a good process

bullTesting is a stepping stone to ldquoprocess improvementrdquoContext-Driven School - emphasizes people setting out to find the bugs that will bemost important to stakeholders

bullSoftware is created by people People set the contextbullTesting finds bugs acting as a skilled mental activitybullKey Question What testing would be most valuable right nowbullExpect changes Adapt testing plans based on test resultsbullTesting research requires empirical and psychological study

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 14110

13 General testing principles

14

bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects

bull Complete exhaustive testing is impossible good strategy and risk managementmust be used

bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs

bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)

bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped

bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)

bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 15110

13 General testing principles ndash heuristics of software testing

bullOperability - The better it works the more efficiently it can be tested

bullObservability - What we see is what we test

bull Controllability - The better we control the software the more the testing processcan be automated and optimized

bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing

bull Simplicity - The less there is to test the more quickly we can test it

bull Stability - The fewer the changes the fewer are the disruptions to testing

bull Understandability - The more information we will have the smarter we will test

bull Suitability - The more we know about the intended use of the software the

better we can organize our testing to find important bugs15

1 4 1 F d t l t t h

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 16110

141 Fundamental test process ndash phases

-Test Planning amp Test control

-Test Analysis amp Design

-Test Implementation ampExecution

-Evaluating exit criteria ampReporting

-Test Closure activities

16

1 4 1 Fundamental test process planning amp control

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 17110

141 Fundamental test process ndash planning amp control

Planning1 Determine scope

bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations

2 Determine risksbull Choose quality risk analysis method (eg FMEA)

bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule

bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies

4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria

Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions

17

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 18110

142 Fundamental test process ndash analysis amp design

bull Reviewing the test basis (such as requirements architecture designinterfaces)

bull Identifying test conditions or test requirements and required test data

based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-

conditionsbull Identify possible test Oracles

bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required

infrastructure and tools

(see Lee Copeland (b2))

18

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 19110

142 Fundamental test process ndash what is a test Oracle

19

bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be

executed The Test Oracle

Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative

hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead

lsquoproblem - no problemrsquo

Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles

Possible issues

bull false alarmsbull missed bugs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 20110

143 Fundamental test process ndash implementation amp execution

Test implementation

bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)

bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission

(see Rex Black (b4))

20

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 21110

143 Fundamental test process ndash priori tizing the Test Cases

Why prioritize the Test Cases

bullIt is not possible to test everything we must do our best in the time available

bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence

bullThis means we must prioritise and focus testing on the priorities

What to watch

bullSeverity of possible defectsbullProbability of possible defects

bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity

21

144 Fundamental test process ndash evaluating exit criteria and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 22110

144 Fundamental test process evaluating exit criteria andreporting

Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed

bull Check if testing mission should be changed

Test report ing

bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity

(see Rex Black (b4) amp RUP- Test discipline(s5))

22

1 4 5 F d l l

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 23110

145 Fundamental test process ndash test closure

bull Verify if test deliverables have been delivered

bull Check and close the remaining active bug reports

bull Archiving the test-ware and environment

bull Handover of the test environment

bull Analyze the identified test process problems (lessons learned)

bull Implement action plan based improvements

(see Rex Black (b4))

23

1 5 Th h l f i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 24110

15 The psychology of testing

24

Testing is regarded as a lsquodestructiversquo activity

(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts

bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise

bullShould be a planned organised kind ofperson

Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices

Rex BlackrsquosTop 10 professional errors

bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations

(see also Brian Marickrsquos article )

1 5 Th h l g f t ti g

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 25110

15 The psychology of testing

25

ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)

ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects

bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs

Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)

Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts

bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence

2 1 1 The V testing model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 26110

211 The V testing model

26

211 The V testing model ndash Verification amp Validationifi i C fi i b

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 5110

112 Why is testing necessary ndash Causes of the errorsDefects are caused by human errors

Why Because ofTime pressure - the more pressure we are under the more likely we are tomake mistakesbullCode complexity or new technology

bullToo many system interactionsbullRequirements not clearly defined changed amp not properly documentedbullWe make wrong assumptions about missing bits of informationbullPoor communicationbullPoor training

5

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 6110

112 Why is testing necessary - Causes of software defects -Defects taxonomy

6

(Boris Beizer)

bull Requirements (incorrect logic completeness verifiability documentation changes)bull Features and functionality (correctness missing case domain and boundarymessages exception mishandled)

bull Structural (control flow sequence data processing)bull Data (definition structure access handling)bull Implementation and Codingbull Integration (internal and external interfaces)bull System (architecture performance recovery partitioning environment)bull Test definition and execution (test design test execution documentation reporting)

(Cem Kaner (b1))bull User interface (functionality communication missing performance output)bull Error handling (prevention detection recovery)bull Boundary (numeric loops)

bull Calculation (wrong constants wrong operation order over amp underflow)bull Initialization (data item string loop control)bull Control flow (stop crash loop if-then-elsehellip)bull Data handling (data type parameter list values)bull Race amp Load conditions (event sequence no resources)bull Source and version control (old bug reappear)bull Testing (fail to notice fail to test fail to report)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 7110

113 Why is testing necessary - The role of testing in thesoftware life cycle

Testers do cooperate with Analysts ndash to review the specifications for completeness and correctnessensure that they are testablebull Designers ndash to improve interfaces testability and usabilitybull Programmers ndash to review the code and assess structural flawsbull Project manager ndash to estimate plan develop test cases perform tests andreport bugs to assess the quality and risksbull Quality assurance staff ndash to provide defect metricsbull Interactions with these project roles are very complex

RACI matrix (Responsible Accountable Consulted Informed)

7

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 8110

114 Why is testing necessary ndash What is quality

8

Quality (ISO) = The totality of the characteristics of an entity that bear on its

ability to satisfy stated or implied needs

bull there are many more definitions

Testing means not only to Verify (the thing is done right)hellip but also to Validate (the right thing is done)

Software quality includes reliability usability efficiency maintainability andportabilityRELIABILITY The ability of the software product to perform its requiredfunctions under stated conditions for a specified period of time or for aspecified number of operationsUSABILITY The capability of the software to be understood learned used

and attractive to the user when used under specified conditionsEFFICIENCY The capability of the software product to provide appropriateperformance relative to the amount of resources used under stated conditionsMAINTAINABILITY The ease with which a software product can be modifiedto correct defects modified to meet new requirements modified to make futuremaintenance easier or adapted to a changed environmentPORTABILITY The ease with which the software product can be transferredfrom one hardware or software environment to another

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 9110

114 Why is testing necessary - Testing and quality

Testing does not inject Quality

into the product but will measurethe level of the Quality of theproduct

Quality can be measured forbull progressbull variance (planned versus actual)

Measuring product quality bullFunctional compliance - functional software requirements testing

bullNon functional requirementsbullTest coverage criteriabullDefect count or defect trend criteria

9

1 1 4 Wh i i li ib

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 10110

114 Why is testing necessary ndash quality attributes

The QUINT model (extended ISO model)

10

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 11110

115 Why is testing necessary - How much testing is enough

when to stop testingThe five basic criteria often used to decide are

bull Previously defined coverage goals have been metbull The defect discovery rate has dropped below a previously defined thresholdbull The cost of finding the next defect exceeds the expected loss from that defectbull The project team reaches consensus that it is appropriate to release the productbull The manager decides to deliver the product

All these criteria are risk basedIt is important not depending on only one stopping criterion

Software Reliability Engineering can help also to determine when to stop testingby taking into consideration aspects like failure intensity

11

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 12110

12 What is testing - Defini tion of testing

12

Testing = the process concerned with planning the necessary static and dynamic

activities preparation and evaluation of software products and related deliverablesin order tobull determine that they satisfy specified requirementsbull demonstrate that they are fit for the intended usebull detect defects help and motivate the developers to fix thembull measure assess and improve the quality of the software product

Testing should be performed throughout the whole software life cycle

There are two basic types of testing execution and non-execution based

Other definitions

bull(IEEE) Testing = the process of analyzing a software item to detect the differencesbetween existing and required conditions and to evaluate its featuresbull(Myers (b3)) Testing = the process of executing a program with the intent of findingerrorsbull(Craig amp Jaskiel (b5)) Testing = a concurrent lifecycle process of engineering usingand maintaining test-ware in order to measure and improve the quality of thesoftware being tested

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 13110

12 What is testing ndash Testing ldquo schoolsrdquo

13

Analytic School - testing is rigorous academic and technicalbullTesting is a branch of CSMathematicsbullTesting techniques must have a logic-mathematical formbullKey Question Which techniques should we usebullRequire precise and detailed specifications

Factory School - testing is a way to measure progress with emphasis on cost and

repeatable standardsbullTesting must be managed amp cost effectivebullTesting validates the product amp measures development progressbullKey Questions How can we measure whether wersquore making progress When will we be donebullRequire clear boundaries between testing and other activities (startstop criteria)

bullEncourage standards (V-model) ldquobest practicesrdquo and certificationQuality School - emphasizes process amp quality acting as the gatekeeper bullSoftware quality requires disciplinebullTesters may need to police developers to follow the rulesbullKey Question Are we following a good process

bullTesting is a stepping stone to ldquoprocess improvementrdquoContext-Driven School - emphasizes people setting out to find the bugs that will bemost important to stakeholders

bullSoftware is created by people People set the contextbullTesting finds bugs acting as a skilled mental activitybullKey Question What testing would be most valuable right nowbullExpect changes Adapt testing plans based on test resultsbullTesting research requires empirical and psychological study

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 14110

13 General testing principles

14

bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects

bull Complete exhaustive testing is impossible good strategy and risk managementmust be used

bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs

bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)

bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped

bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)

bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 15110

13 General testing principles ndash heuristics of software testing

bullOperability - The better it works the more efficiently it can be tested

bullObservability - What we see is what we test

bull Controllability - The better we control the software the more the testing processcan be automated and optimized

bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing

bull Simplicity - The less there is to test the more quickly we can test it

bull Stability - The fewer the changes the fewer are the disruptions to testing

bull Understandability - The more information we will have the smarter we will test

bull Suitability - The more we know about the intended use of the software the

better we can organize our testing to find important bugs15

1 4 1 F d t l t t h

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 16110

141 Fundamental test process ndash phases

-Test Planning amp Test control

-Test Analysis amp Design

-Test Implementation ampExecution

-Evaluating exit criteria ampReporting

-Test Closure activities

16

1 4 1 Fundamental test process planning amp control

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 17110

141 Fundamental test process ndash planning amp control

Planning1 Determine scope

bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations

2 Determine risksbull Choose quality risk analysis method (eg FMEA)

bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule

bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies

4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria

Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions

17

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 18110

142 Fundamental test process ndash analysis amp design

bull Reviewing the test basis (such as requirements architecture designinterfaces)

bull Identifying test conditions or test requirements and required test data

based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-

conditionsbull Identify possible test Oracles

bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required

infrastructure and tools

(see Lee Copeland (b2))

18

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 19110

142 Fundamental test process ndash what is a test Oracle

19

bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be

executed The Test Oracle

Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative

hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead

lsquoproblem - no problemrsquo

Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles

Possible issues

bull false alarmsbull missed bugs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 20110

143 Fundamental test process ndash implementation amp execution

Test implementation

bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)

bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission

(see Rex Black (b4))

20

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 21110

143 Fundamental test process ndash priori tizing the Test Cases

Why prioritize the Test Cases

bullIt is not possible to test everything we must do our best in the time available

bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence

bullThis means we must prioritise and focus testing on the priorities

What to watch

bullSeverity of possible defectsbullProbability of possible defects

bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity

21

144 Fundamental test process ndash evaluating exit criteria and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 22110

144 Fundamental test process evaluating exit criteria andreporting

Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed

bull Check if testing mission should be changed

Test report ing

bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity

(see Rex Black (b4) amp RUP- Test discipline(s5))

22

1 4 5 F d l l

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 23110

145 Fundamental test process ndash test closure

bull Verify if test deliverables have been delivered

bull Check and close the remaining active bug reports

bull Archiving the test-ware and environment

bull Handover of the test environment

bull Analyze the identified test process problems (lessons learned)

bull Implement action plan based improvements

(see Rex Black (b4))

23

1 5 Th h l f i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 24110

15 The psychology of testing

24

Testing is regarded as a lsquodestructiversquo activity

(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts

bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise

bullShould be a planned organised kind ofperson

Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices

Rex BlackrsquosTop 10 professional errors

bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations

(see also Brian Marickrsquos article )

1 5 Th h l g f t ti g

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 25110

15 The psychology of testing

25

ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)

ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects

bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs

Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)

Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts

bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence

2 1 1 The V testing model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 26110

211 The V testing model

26

211 The V testing model ndash Verification amp Validationifi i C fi i b

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 6110

112 Why is testing necessary - Causes of software defects -Defects taxonomy

6

(Boris Beizer)

bull Requirements (incorrect logic completeness verifiability documentation changes)bull Features and functionality (correctness missing case domain and boundarymessages exception mishandled)

bull Structural (control flow sequence data processing)bull Data (definition structure access handling)bull Implementation and Codingbull Integration (internal and external interfaces)bull System (architecture performance recovery partitioning environment)bull Test definition and execution (test design test execution documentation reporting)

(Cem Kaner (b1))bull User interface (functionality communication missing performance output)bull Error handling (prevention detection recovery)bull Boundary (numeric loops)

bull Calculation (wrong constants wrong operation order over amp underflow)bull Initialization (data item string loop control)bull Control flow (stop crash loop if-then-elsehellip)bull Data handling (data type parameter list values)bull Race amp Load conditions (event sequence no resources)bull Source and version control (old bug reappear)bull Testing (fail to notice fail to test fail to report)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 7110

113 Why is testing necessary - The role of testing in thesoftware life cycle

Testers do cooperate with Analysts ndash to review the specifications for completeness and correctnessensure that they are testablebull Designers ndash to improve interfaces testability and usabilitybull Programmers ndash to review the code and assess structural flawsbull Project manager ndash to estimate plan develop test cases perform tests andreport bugs to assess the quality and risksbull Quality assurance staff ndash to provide defect metricsbull Interactions with these project roles are very complex

RACI matrix (Responsible Accountable Consulted Informed)

7

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 8110

114 Why is testing necessary ndash What is quality

8

Quality (ISO) = The totality of the characteristics of an entity that bear on its

ability to satisfy stated or implied needs

bull there are many more definitions

Testing means not only to Verify (the thing is done right)hellip but also to Validate (the right thing is done)

Software quality includes reliability usability efficiency maintainability andportabilityRELIABILITY The ability of the software product to perform its requiredfunctions under stated conditions for a specified period of time or for aspecified number of operationsUSABILITY The capability of the software to be understood learned used

and attractive to the user when used under specified conditionsEFFICIENCY The capability of the software product to provide appropriateperformance relative to the amount of resources used under stated conditionsMAINTAINABILITY The ease with which a software product can be modifiedto correct defects modified to meet new requirements modified to make futuremaintenance easier or adapted to a changed environmentPORTABILITY The ease with which the software product can be transferredfrom one hardware or software environment to another

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 9110

114 Why is testing necessary - Testing and quality

Testing does not inject Quality

into the product but will measurethe level of the Quality of theproduct

Quality can be measured forbull progressbull variance (planned versus actual)

Measuring product quality bullFunctional compliance - functional software requirements testing

bullNon functional requirementsbullTest coverage criteriabullDefect count or defect trend criteria

9

1 1 4 Wh i i li ib

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 10110

114 Why is testing necessary ndash quality attributes

The QUINT model (extended ISO model)

10

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 11110

115 Why is testing necessary - How much testing is enough

when to stop testingThe five basic criteria often used to decide are

bull Previously defined coverage goals have been metbull The defect discovery rate has dropped below a previously defined thresholdbull The cost of finding the next defect exceeds the expected loss from that defectbull The project team reaches consensus that it is appropriate to release the productbull The manager decides to deliver the product

All these criteria are risk basedIt is important not depending on only one stopping criterion

Software Reliability Engineering can help also to determine when to stop testingby taking into consideration aspects like failure intensity

11

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 12110

12 What is testing - Defini tion of testing

12

Testing = the process concerned with planning the necessary static and dynamic

activities preparation and evaluation of software products and related deliverablesin order tobull determine that they satisfy specified requirementsbull demonstrate that they are fit for the intended usebull detect defects help and motivate the developers to fix thembull measure assess and improve the quality of the software product

Testing should be performed throughout the whole software life cycle

There are two basic types of testing execution and non-execution based

Other definitions

bull(IEEE) Testing = the process of analyzing a software item to detect the differencesbetween existing and required conditions and to evaluate its featuresbull(Myers (b3)) Testing = the process of executing a program with the intent of findingerrorsbull(Craig amp Jaskiel (b5)) Testing = a concurrent lifecycle process of engineering usingand maintaining test-ware in order to measure and improve the quality of thesoftware being tested

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 13110

12 What is testing ndash Testing ldquo schoolsrdquo

13

Analytic School - testing is rigorous academic and technicalbullTesting is a branch of CSMathematicsbullTesting techniques must have a logic-mathematical formbullKey Question Which techniques should we usebullRequire precise and detailed specifications

Factory School - testing is a way to measure progress with emphasis on cost and

repeatable standardsbullTesting must be managed amp cost effectivebullTesting validates the product amp measures development progressbullKey Questions How can we measure whether wersquore making progress When will we be donebullRequire clear boundaries between testing and other activities (startstop criteria)

bullEncourage standards (V-model) ldquobest practicesrdquo and certificationQuality School - emphasizes process amp quality acting as the gatekeeper bullSoftware quality requires disciplinebullTesters may need to police developers to follow the rulesbullKey Question Are we following a good process

bullTesting is a stepping stone to ldquoprocess improvementrdquoContext-Driven School - emphasizes people setting out to find the bugs that will bemost important to stakeholders

bullSoftware is created by people People set the contextbullTesting finds bugs acting as a skilled mental activitybullKey Question What testing would be most valuable right nowbullExpect changes Adapt testing plans based on test resultsbullTesting research requires empirical and psychological study

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 14110

13 General testing principles

14

bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects

bull Complete exhaustive testing is impossible good strategy and risk managementmust be used

bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs

bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)

bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped

bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)

bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 15110

13 General testing principles ndash heuristics of software testing

bullOperability - The better it works the more efficiently it can be tested

bullObservability - What we see is what we test

bull Controllability - The better we control the software the more the testing processcan be automated and optimized

bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing

bull Simplicity - The less there is to test the more quickly we can test it

bull Stability - The fewer the changes the fewer are the disruptions to testing

bull Understandability - The more information we will have the smarter we will test

bull Suitability - The more we know about the intended use of the software the

better we can organize our testing to find important bugs15

1 4 1 F d t l t t h

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 16110

141 Fundamental test process ndash phases

-Test Planning amp Test control

-Test Analysis amp Design

-Test Implementation ampExecution

-Evaluating exit criteria ampReporting

-Test Closure activities

16

1 4 1 Fundamental test process planning amp control

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 17110

141 Fundamental test process ndash planning amp control

Planning1 Determine scope

bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations

2 Determine risksbull Choose quality risk analysis method (eg FMEA)

bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule

bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies

4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria

Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions

17

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 18110

142 Fundamental test process ndash analysis amp design

bull Reviewing the test basis (such as requirements architecture designinterfaces)

bull Identifying test conditions or test requirements and required test data

based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-

conditionsbull Identify possible test Oracles

bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required

infrastructure and tools

(see Lee Copeland (b2))

18

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 19110

142 Fundamental test process ndash what is a test Oracle

19

bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be

executed The Test Oracle

Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative

hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead

lsquoproblem - no problemrsquo

Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles

Possible issues

bull false alarmsbull missed bugs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 20110

143 Fundamental test process ndash implementation amp execution

Test implementation

bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)

bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission

(see Rex Black (b4))

20

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 21110

143 Fundamental test process ndash priori tizing the Test Cases

Why prioritize the Test Cases

bullIt is not possible to test everything we must do our best in the time available

bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence

bullThis means we must prioritise and focus testing on the priorities

What to watch

bullSeverity of possible defectsbullProbability of possible defects

bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity

21

144 Fundamental test process ndash evaluating exit criteria and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 22110

144 Fundamental test process evaluating exit criteria andreporting

Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed

bull Check if testing mission should be changed

Test report ing

bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity

(see Rex Black (b4) amp RUP- Test discipline(s5))

22

1 4 5 F d l l

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 23110

145 Fundamental test process ndash test closure

bull Verify if test deliverables have been delivered

bull Check and close the remaining active bug reports

bull Archiving the test-ware and environment

bull Handover of the test environment

bull Analyze the identified test process problems (lessons learned)

bull Implement action plan based improvements

(see Rex Black (b4))

23

1 5 Th h l f i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 24110

15 The psychology of testing

24

Testing is regarded as a lsquodestructiversquo activity

(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts

bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise

bullShould be a planned organised kind ofperson

Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices

Rex BlackrsquosTop 10 professional errors

bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations

(see also Brian Marickrsquos article )

1 5 Th h l g f t ti g

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 25110

15 The psychology of testing

25

ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)

ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects

bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs

Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)

Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts

bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence

2 1 1 The V testing model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 26110

211 The V testing model

26

211 The V testing model ndash Verification amp Validationifi i C fi i b

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 7110

113 Why is testing necessary - The role of testing in thesoftware life cycle

Testers do cooperate with Analysts ndash to review the specifications for completeness and correctnessensure that they are testablebull Designers ndash to improve interfaces testability and usabilitybull Programmers ndash to review the code and assess structural flawsbull Project manager ndash to estimate plan develop test cases perform tests andreport bugs to assess the quality and risksbull Quality assurance staff ndash to provide defect metricsbull Interactions with these project roles are very complex

RACI matrix (Responsible Accountable Consulted Informed)

7

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 8110

114 Why is testing necessary ndash What is quality

8

Quality (ISO) = The totality of the characteristics of an entity that bear on its

ability to satisfy stated or implied needs

bull there are many more definitions

Testing means not only to Verify (the thing is done right)hellip but also to Validate (the right thing is done)

Software quality includes reliability usability efficiency maintainability andportabilityRELIABILITY The ability of the software product to perform its requiredfunctions under stated conditions for a specified period of time or for aspecified number of operationsUSABILITY The capability of the software to be understood learned used

and attractive to the user when used under specified conditionsEFFICIENCY The capability of the software product to provide appropriateperformance relative to the amount of resources used under stated conditionsMAINTAINABILITY The ease with which a software product can be modifiedto correct defects modified to meet new requirements modified to make futuremaintenance easier or adapted to a changed environmentPORTABILITY The ease with which the software product can be transferredfrom one hardware or software environment to another

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 9110

114 Why is testing necessary - Testing and quality

Testing does not inject Quality

into the product but will measurethe level of the Quality of theproduct

Quality can be measured forbull progressbull variance (planned versus actual)

Measuring product quality bullFunctional compliance - functional software requirements testing

bullNon functional requirementsbullTest coverage criteriabullDefect count or defect trend criteria

9

1 1 4 Wh i i li ib

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 10110

114 Why is testing necessary ndash quality attributes

The QUINT model (extended ISO model)

10

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 11110

115 Why is testing necessary - How much testing is enough

when to stop testingThe five basic criteria often used to decide are

bull Previously defined coverage goals have been metbull The defect discovery rate has dropped below a previously defined thresholdbull The cost of finding the next defect exceeds the expected loss from that defectbull The project team reaches consensus that it is appropriate to release the productbull The manager decides to deliver the product

All these criteria are risk basedIt is important not depending on only one stopping criterion

Software Reliability Engineering can help also to determine when to stop testingby taking into consideration aspects like failure intensity

11

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 12110

12 What is testing - Defini tion of testing

12

Testing = the process concerned with planning the necessary static and dynamic

activities preparation and evaluation of software products and related deliverablesin order tobull determine that they satisfy specified requirementsbull demonstrate that they are fit for the intended usebull detect defects help and motivate the developers to fix thembull measure assess and improve the quality of the software product

Testing should be performed throughout the whole software life cycle

There are two basic types of testing execution and non-execution based

Other definitions

bull(IEEE) Testing = the process of analyzing a software item to detect the differencesbetween existing and required conditions and to evaluate its featuresbull(Myers (b3)) Testing = the process of executing a program with the intent of findingerrorsbull(Craig amp Jaskiel (b5)) Testing = a concurrent lifecycle process of engineering usingand maintaining test-ware in order to measure and improve the quality of thesoftware being tested

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 13110

12 What is testing ndash Testing ldquo schoolsrdquo

13

Analytic School - testing is rigorous academic and technicalbullTesting is a branch of CSMathematicsbullTesting techniques must have a logic-mathematical formbullKey Question Which techniques should we usebullRequire precise and detailed specifications

Factory School - testing is a way to measure progress with emphasis on cost and

repeatable standardsbullTesting must be managed amp cost effectivebullTesting validates the product amp measures development progressbullKey Questions How can we measure whether wersquore making progress When will we be donebullRequire clear boundaries between testing and other activities (startstop criteria)

bullEncourage standards (V-model) ldquobest practicesrdquo and certificationQuality School - emphasizes process amp quality acting as the gatekeeper bullSoftware quality requires disciplinebullTesters may need to police developers to follow the rulesbullKey Question Are we following a good process

bullTesting is a stepping stone to ldquoprocess improvementrdquoContext-Driven School - emphasizes people setting out to find the bugs that will bemost important to stakeholders

bullSoftware is created by people People set the contextbullTesting finds bugs acting as a skilled mental activitybullKey Question What testing would be most valuable right nowbullExpect changes Adapt testing plans based on test resultsbullTesting research requires empirical and psychological study

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 14110

13 General testing principles

14

bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects

bull Complete exhaustive testing is impossible good strategy and risk managementmust be used

bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs

bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)

bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped

bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)

bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 15110

13 General testing principles ndash heuristics of software testing

bullOperability - The better it works the more efficiently it can be tested

bullObservability - What we see is what we test

bull Controllability - The better we control the software the more the testing processcan be automated and optimized

bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing

bull Simplicity - The less there is to test the more quickly we can test it

bull Stability - The fewer the changes the fewer are the disruptions to testing

bull Understandability - The more information we will have the smarter we will test

bull Suitability - The more we know about the intended use of the software the

better we can organize our testing to find important bugs15

1 4 1 F d t l t t h

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 16110

141 Fundamental test process ndash phases

-Test Planning amp Test control

-Test Analysis amp Design

-Test Implementation ampExecution

-Evaluating exit criteria ampReporting

-Test Closure activities

16

1 4 1 Fundamental test process planning amp control

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 17110

141 Fundamental test process ndash planning amp control

Planning1 Determine scope

bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations

2 Determine risksbull Choose quality risk analysis method (eg FMEA)

bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule

bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies

4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria

Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions

17

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 18110

142 Fundamental test process ndash analysis amp design

bull Reviewing the test basis (such as requirements architecture designinterfaces)

bull Identifying test conditions or test requirements and required test data

based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-

conditionsbull Identify possible test Oracles

bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required

infrastructure and tools

(see Lee Copeland (b2))

18

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 19110

142 Fundamental test process ndash what is a test Oracle

19

bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be

executed The Test Oracle

Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative

hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead

lsquoproblem - no problemrsquo

Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles

Possible issues

bull false alarmsbull missed bugs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 20110

143 Fundamental test process ndash implementation amp execution

Test implementation

bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)

bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission

(see Rex Black (b4))

20

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 21110

143 Fundamental test process ndash priori tizing the Test Cases

Why prioritize the Test Cases

bullIt is not possible to test everything we must do our best in the time available

bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence

bullThis means we must prioritise and focus testing on the priorities

What to watch

bullSeverity of possible defectsbullProbability of possible defects

bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity

21

144 Fundamental test process ndash evaluating exit criteria and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 22110

144 Fundamental test process evaluating exit criteria andreporting

Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed

bull Check if testing mission should be changed

Test report ing

bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity

(see Rex Black (b4) amp RUP- Test discipline(s5))

22

1 4 5 F d l l

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 23110

145 Fundamental test process ndash test closure

bull Verify if test deliverables have been delivered

bull Check and close the remaining active bug reports

bull Archiving the test-ware and environment

bull Handover of the test environment

bull Analyze the identified test process problems (lessons learned)

bull Implement action plan based improvements

(see Rex Black (b4))

23

1 5 Th h l f i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 24110

15 The psychology of testing

24

Testing is regarded as a lsquodestructiversquo activity

(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts

bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise

bullShould be a planned organised kind ofperson

Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices

Rex BlackrsquosTop 10 professional errors

bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations

(see also Brian Marickrsquos article )

1 5 Th h l g f t ti g

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 25110

15 The psychology of testing

25

ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)

ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects

bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs

Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)

Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts

bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence

2 1 1 The V testing model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 26110

211 The V testing model

26

211 The V testing model ndash Verification amp Validationifi i C fi i b

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 8110

114 Why is testing necessary ndash What is quality

8

Quality (ISO) = The totality of the characteristics of an entity that bear on its

ability to satisfy stated or implied needs

bull there are many more definitions

Testing means not only to Verify (the thing is done right)hellip but also to Validate (the right thing is done)

Software quality includes reliability usability efficiency maintainability andportabilityRELIABILITY The ability of the software product to perform its requiredfunctions under stated conditions for a specified period of time or for aspecified number of operationsUSABILITY The capability of the software to be understood learned used

and attractive to the user when used under specified conditionsEFFICIENCY The capability of the software product to provide appropriateperformance relative to the amount of resources used under stated conditionsMAINTAINABILITY The ease with which a software product can be modifiedto correct defects modified to meet new requirements modified to make futuremaintenance easier or adapted to a changed environmentPORTABILITY The ease with which the software product can be transferredfrom one hardware or software environment to another

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 9110

114 Why is testing necessary - Testing and quality

Testing does not inject Quality

into the product but will measurethe level of the Quality of theproduct

Quality can be measured forbull progressbull variance (planned versus actual)

Measuring product quality bullFunctional compliance - functional software requirements testing

bullNon functional requirementsbullTest coverage criteriabullDefect count or defect trend criteria

9

1 1 4 Wh i i li ib

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 10110

114 Why is testing necessary ndash quality attributes

The QUINT model (extended ISO model)

10

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 11110

115 Why is testing necessary - How much testing is enough

when to stop testingThe five basic criteria often used to decide are

bull Previously defined coverage goals have been metbull The defect discovery rate has dropped below a previously defined thresholdbull The cost of finding the next defect exceeds the expected loss from that defectbull The project team reaches consensus that it is appropriate to release the productbull The manager decides to deliver the product

All these criteria are risk basedIt is important not depending on only one stopping criterion

Software Reliability Engineering can help also to determine when to stop testingby taking into consideration aspects like failure intensity

11

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 12110

12 What is testing - Defini tion of testing

12

Testing = the process concerned with planning the necessary static and dynamic

activities preparation and evaluation of software products and related deliverablesin order tobull determine that they satisfy specified requirementsbull demonstrate that they are fit for the intended usebull detect defects help and motivate the developers to fix thembull measure assess and improve the quality of the software product

Testing should be performed throughout the whole software life cycle

There are two basic types of testing execution and non-execution based

Other definitions

bull(IEEE) Testing = the process of analyzing a software item to detect the differencesbetween existing and required conditions and to evaluate its featuresbull(Myers (b3)) Testing = the process of executing a program with the intent of findingerrorsbull(Craig amp Jaskiel (b5)) Testing = a concurrent lifecycle process of engineering usingand maintaining test-ware in order to measure and improve the quality of thesoftware being tested

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 13110

12 What is testing ndash Testing ldquo schoolsrdquo

13

Analytic School - testing is rigorous academic and technicalbullTesting is a branch of CSMathematicsbullTesting techniques must have a logic-mathematical formbullKey Question Which techniques should we usebullRequire precise and detailed specifications

Factory School - testing is a way to measure progress with emphasis on cost and

repeatable standardsbullTesting must be managed amp cost effectivebullTesting validates the product amp measures development progressbullKey Questions How can we measure whether wersquore making progress When will we be donebullRequire clear boundaries between testing and other activities (startstop criteria)

bullEncourage standards (V-model) ldquobest practicesrdquo and certificationQuality School - emphasizes process amp quality acting as the gatekeeper bullSoftware quality requires disciplinebullTesters may need to police developers to follow the rulesbullKey Question Are we following a good process

bullTesting is a stepping stone to ldquoprocess improvementrdquoContext-Driven School - emphasizes people setting out to find the bugs that will bemost important to stakeholders

bullSoftware is created by people People set the contextbullTesting finds bugs acting as a skilled mental activitybullKey Question What testing would be most valuable right nowbullExpect changes Adapt testing plans based on test resultsbullTesting research requires empirical and psychological study

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 14110

13 General testing principles

14

bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects

bull Complete exhaustive testing is impossible good strategy and risk managementmust be used

bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs

bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)

bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped

bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)

bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 15110

13 General testing principles ndash heuristics of software testing

bullOperability - The better it works the more efficiently it can be tested

bullObservability - What we see is what we test

bull Controllability - The better we control the software the more the testing processcan be automated and optimized

bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing

bull Simplicity - The less there is to test the more quickly we can test it

bull Stability - The fewer the changes the fewer are the disruptions to testing

bull Understandability - The more information we will have the smarter we will test

bull Suitability - The more we know about the intended use of the software the

better we can organize our testing to find important bugs15

1 4 1 F d t l t t h

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 16110

141 Fundamental test process ndash phases

-Test Planning amp Test control

-Test Analysis amp Design

-Test Implementation ampExecution

-Evaluating exit criteria ampReporting

-Test Closure activities

16

1 4 1 Fundamental test process planning amp control

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 17110

141 Fundamental test process ndash planning amp control

Planning1 Determine scope

bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations

2 Determine risksbull Choose quality risk analysis method (eg FMEA)

bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule

bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies

4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria

Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions

17

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 18110

142 Fundamental test process ndash analysis amp design

bull Reviewing the test basis (such as requirements architecture designinterfaces)

bull Identifying test conditions or test requirements and required test data

based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-

conditionsbull Identify possible test Oracles

bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required

infrastructure and tools

(see Lee Copeland (b2))

18

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 19110

142 Fundamental test process ndash what is a test Oracle

19

bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be

executed The Test Oracle

Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative

hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead

lsquoproblem - no problemrsquo

Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles

Possible issues

bull false alarmsbull missed bugs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 20110

143 Fundamental test process ndash implementation amp execution

Test implementation

bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)

bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission

(see Rex Black (b4))

20

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 21110

143 Fundamental test process ndash priori tizing the Test Cases

Why prioritize the Test Cases

bullIt is not possible to test everything we must do our best in the time available

bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence

bullThis means we must prioritise and focus testing on the priorities

What to watch

bullSeverity of possible defectsbullProbability of possible defects

bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity

21

144 Fundamental test process ndash evaluating exit criteria and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 22110

144 Fundamental test process evaluating exit criteria andreporting

Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed

bull Check if testing mission should be changed

Test report ing

bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity

(see Rex Black (b4) amp RUP- Test discipline(s5))

22

1 4 5 F d l l

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 23110

145 Fundamental test process ndash test closure

bull Verify if test deliverables have been delivered

bull Check and close the remaining active bug reports

bull Archiving the test-ware and environment

bull Handover of the test environment

bull Analyze the identified test process problems (lessons learned)

bull Implement action plan based improvements

(see Rex Black (b4))

23

1 5 Th h l f i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 24110

15 The psychology of testing

24

Testing is regarded as a lsquodestructiversquo activity

(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts

bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise

bullShould be a planned organised kind ofperson

Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices

Rex BlackrsquosTop 10 professional errors

bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations

(see also Brian Marickrsquos article )

1 5 Th h l g f t ti g

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 25110

15 The psychology of testing

25

ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)

ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects

bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs

Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)

Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts

bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence

2 1 1 The V testing model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 26110

211 The V testing model

26

211 The V testing model ndash Verification amp Validationifi i C fi i b

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 9110

114 Why is testing necessary - Testing and quality

Testing does not inject Quality

into the product but will measurethe level of the Quality of theproduct

Quality can be measured forbull progressbull variance (planned versus actual)

Measuring product quality bullFunctional compliance - functional software requirements testing

bullNon functional requirementsbullTest coverage criteriabullDefect count or defect trend criteria

9

1 1 4 Wh i i li ib

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 10110

114 Why is testing necessary ndash quality attributes

The QUINT model (extended ISO model)

10

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 11110

115 Why is testing necessary - How much testing is enough

when to stop testingThe five basic criteria often used to decide are

bull Previously defined coverage goals have been metbull The defect discovery rate has dropped below a previously defined thresholdbull The cost of finding the next defect exceeds the expected loss from that defectbull The project team reaches consensus that it is appropriate to release the productbull The manager decides to deliver the product

All these criteria are risk basedIt is important not depending on only one stopping criterion

Software Reliability Engineering can help also to determine when to stop testingby taking into consideration aspects like failure intensity

11

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 12110

12 What is testing - Defini tion of testing

12

Testing = the process concerned with planning the necessary static and dynamic

activities preparation and evaluation of software products and related deliverablesin order tobull determine that they satisfy specified requirementsbull demonstrate that they are fit for the intended usebull detect defects help and motivate the developers to fix thembull measure assess and improve the quality of the software product

Testing should be performed throughout the whole software life cycle

There are two basic types of testing execution and non-execution based

Other definitions

bull(IEEE) Testing = the process of analyzing a software item to detect the differencesbetween existing and required conditions and to evaluate its featuresbull(Myers (b3)) Testing = the process of executing a program with the intent of findingerrorsbull(Craig amp Jaskiel (b5)) Testing = a concurrent lifecycle process of engineering usingand maintaining test-ware in order to measure and improve the quality of thesoftware being tested

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 13110

12 What is testing ndash Testing ldquo schoolsrdquo

13

Analytic School - testing is rigorous academic and technicalbullTesting is a branch of CSMathematicsbullTesting techniques must have a logic-mathematical formbullKey Question Which techniques should we usebullRequire precise and detailed specifications

Factory School - testing is a way to measure progress with emphasis on cost and

repeatable standardsbullTesting must be managed amp cost effectivebullTesting validates the product amp measures development progressbullKey Questions How can we measure whether wersquore making progress When will we be donebullRequire clear boundaries between testing and other activities (startstop criteria)

bullEncourage standards (V-model) ldquobest practicesrdquo and certificationQuality School - emphasizes process amp quality acting as the gatekeeper bullSoftware quality requires disciplinebullTesters may need to police developers to follow the rulesbullKey Question Are we following a good process

bullTesting is a stepping stone to ldquoprocess improvementrdquoContext-Driven School - emphasizes people setting out to find the bugs that will bemost important to stakeholders

bullSoftware is created by people People set the contextbullTesting finds bugs acting as a skilled mental activitybullKey Question What testing would be most valuable right nowbullExpect changes Adapt testing plans based on test resultsbullTesting research requires empirical and psychological study

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 14110

13 General testing principles

14

bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects

bull Complete exhaustive testing is impossible good strategy and risk managementmust be used

bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs

bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)

bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped

bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)

bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 15110

13 General testing principles ndash heuristics of software testing

bullOperability - The better it works the more efficiently it can be tested

bullObservability - What we see is what we test

bull Controllability - The better we control the software the more the testing processcan be automated and optimized

bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing

bull Simplicity - The less there is to test the more quickly we can test it

bull Stability - The fewer the changes the fewer are the disruptions to testing

bull Understandability - The more information we will have the smarter we will test

bull Suitability - The more we know about the intended use of the software the

better we can organize our testing to find important bugs15

1 4 1 F d t l t t h

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 16110

141 Fundamental test process ndash phases

-Test Planning amp Test control

-Test Analysis amp Design

-Test Implementation ampExecution

-Evaluating exit criteria ampReporting

-Test Closure activities

16

1 4 1 Fundamental test process planning amp control

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 17110

141 Fundamental test process ndash planning amp control

Planning1 Determine scope

bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations

2 Determine risksbull Choose quality risk analysis method (eg FMEA)

bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule

bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies

4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria

Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions

17

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 18110

142 Fundamental test process ndash analysis amp design

bull Reviewing the test basis (such as requirements architecture designinterfaces)

bull Identifying test conditions or test requirements and required test data

based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-

conditionsbull Identify possible test Oracles

bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required

infrastructure and tools

(see Lee Copeland (b2))

18

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 19110

142 Fundamental test process ndash what is a test Oracle

19

bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be

executed The Test Oracle

Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative

hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead

lsquoproblem - no problemrsquo

Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles

Possible issues

bull false alarmsbull missed bugs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 20110

143 Fundamental test process ndash implementation amp execution

Test implementation

bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)

bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission

(see Rex Black (b4))

20

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 21110

143 Fundamental test process ndash priori tizing the Test Cases

Why prioritize the Test Cases

bullIt is not possible to test everything we must do our best in the time available

bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence

bullThis means we must prioritise and focus testing on the priorities

What to watch

bullSeverity of possible defectsbullProbability of possible defects

bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity

21

144 Fundamental test process ndash evaluating exit criteria and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 22110

144 Fundamental test process evaluating exit criteria andreporting

Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed

bull Check if testing mission should be changed

Test report ing

bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity

(see Rex Black (b4) amp RUP- Test discipline(s5))

22

1 4 5 F d l l

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 23110

145 Fundamental test process ndash test closure

bull Verify if test deliverables have been delivered

bull Check and close the remaining active bug reports

bull Archiving the test-ware and environment

bull Handover of the test environment

bull Analyze the identified test process problems (lessons learned)

bull Implement action plan based improvements

(see Rex Black (b4))

23

1 5 Th h l f i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 24110

15 The psychology of testing

24

Testing is regarded as a lsquodestructiversquo activity

(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts

bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise

bullShould be a planned organised kind ofperson

Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices

Rex BlackrsquosTop 10 professional errors

bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations

(see also Brian Marickrsquos article )

1 5 Th h l g f t ti g

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 25110

15 The psychology of testing

25

ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)

ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects

bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs

Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)

Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts

bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence

2 1 1 The V testing model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 26110

211 The V testing model

26

211 The V testing model ndash Verification amp Validationifi i C fi i b

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 10110

114 Why is testing necessary ndash quality attributes

The QUINT model (extended ISO model)

10

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 11110

115 Why is testing necessary - How much testing is enough

when to stop testingThe five basic criteria often used to decide are

bull Previously defined coverage goals have been metbull The defect discovery rate has dropped below a previously defined thresholdbull The cost of finding the next defect exceeds the expected loss from that defectbull The project team reaches consensus that it is appropriate to release the productbull The manager decides to deliver the product

All these criteria are risk basedIt is important not depending on only one stopping criterion

Software Reliability Engineering can help also to determine when to stop testingby taking into consideration aspects like failure intensity

11

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 12110

12 What is testing - Defini tion of testing

12

Testing = the process concerned with planning the necessary static and dynamic

activities preparation and evaluation of software products and related deliverablesin order tobull determine that they satisfy specified requirementsbull demonstrate that they are fit for the intended usebull detect defects help and motivate the developers to fix thembull measure assess and improve the quality of the software product

Testing should be performed throughout the whole software life cycle

There are two basic types of testing execution and non-execution based

Other definitions

bull(IEEE) Testing = the process of analyzing a software item to detect the differencesbetween existing and required conditions and to evaluate its featuresbull(Myers (b3)) Testing = the process of executing a program with the intent of findingerrorsbull(Craig amp Jaskiel (b5)) Testing = a concurrent lifecycle process of engineering usingand maintaining test-ware in order to measure and improve the quality of thesoftware being tested

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 13110

12 What is testing ndash Testing ldquo schoolsrdquo

13

Analytic School - testing is rigorous academic and technicalbullTesting is a branch of CSMathematicsbullTesting techniques must have a logic-mathematical formbullKey Question Which techniques should we usebullRequire precise and detailed specifications

Factory School - testing is a way to measure progress with emphasis on cost and

repeatable standardsbullTesting must be managed amp cost effectivebullTesting validates the product amp measures development progressbullKey Questions How can we measure whether wersquore making progress When will we be donebullRequire clear boundaries between testing and other activities (startstop criteria)

bullEncourage standards (V-model) ldquobest practicesrdquo and certificationQuality School - emphasizes process amp quality acting as the gatekeeper bullSoftware quality requires disciplinebullTesters may need to police developers to follow the rulesbullKey Question Are we following a good process

bullTesting is a stepping stone to ldquoprocess improvementrdquoContext-Driven School - emphasizes people setting out to find the bugs that will bemost important to stakeholders

bullSoftware is created by people People set the contextbullTesting finds bugs acting as a skilled mental activitybullKey Question What testing would be most valuable right nowbullExpect changes Adapt testing plans based on test resultsbullTesting research requires empirical and psychological study

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 14110

13 General testing principles

14

bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects

bull Complete exhaustive testing is impossible good strategy and risk managementmust be used

bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs

bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)

bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped

bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)

bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 15110

13 General testing principles ndash heuristics of software testing

bullOperability - The better it works the more efficiently it can be tested

bullObservability - What we see is what we test

bull Controllability - The better we control the software the more the testing processcan be automated and optimized

bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing

bull Simplicity - The less there is to test the more quickly we can test it

bull Stability - The fewer the changes the fewer are the disruptions to testing

bull Understandability - The more information we will have the smarter we will test

bull Suitability - The more we know about the intended use of the software the

better we can organize our testing to find important bugs15

1 4 1 F d t l t t h

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 16110

141 Fundamental test process ndash phases

-Test Planning amp Test control

-Test Analysis amp Design

-Test Implementation ampExecution

-Evaluating exit criteria ampReporting

-Test Closure activities

16

1 4 1 Fundamental test process planning amp control

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 17110

141 Fundamental test process ndash planning amp control

Planning1 Determine scope

bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations

2 Determine risksbull Choose quality risk analysis method (eg FMEA)

bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule

bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies

4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria

Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions

17

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 18110

142 Fundamental test process ndash analysis amp design

bull Reviewing the test basis (such as requirements architecture designinterfaces)

bull Identifying test conditions or test requirements and required test data

based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-

conditionsbull Identify possible test Oracles

bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required

infrastructure and tools

(see Lee Copeland (b2))

18

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 19110

142 Fundamental test process ndash what is a test Oracle

19

bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be

executed The Test Oracle

Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative

hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead

lsquoproblem - no problemrsquo

Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles

Possible issues

bull false alarmsbull missed bugs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 20110

143 Fundamental test process ndash implementation amp execution

Test implementation

bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)

bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission

(see Rex Black (b4))

20

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 21110

143 Fundamental test process ndash priori tizing the Test Cases

Why prioritize the Test Cases

bullIt is not possible to test everything we must do our best in the time available

bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence

bullThis means we must prioritise and focus testing on the priorities

What to watch

bullSeverity of possible defectsbullProbability of possible defects

bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity

21

144 Fundamental test process ndash evaluating exit criteria and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 22110

144 Fundamental test process evaluating exit criteria andreporting

Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed

bull Check if testing mission should be changed

Test report ing

bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity

(see Rex Black (b4) amp RUP- Test discipline(s5))

22

1 4 5 F d l l

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 23110

145 Fundamental test process ndash test closure

bull Verify if test deliverables have been delivered

bull Check and close the remaining active bug reports

bull Archiving the test-ware and environment

bull Handover of the test environment

bull Analyze the identified test process problems (lessons learned)

bull Implement action plan based improvements

(see Rex Black (b4))

23

1 5 Th h l f i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 24110

15 The psychology of testing

24

Testing is regarded as a lsquodestructiversquo activity

(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts

bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise

bullShould be a planned organised kind ofperson

Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices

Rex BlackrsquosTop 10 professional errors

bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations

(see also Brian Marickrsquos article )

1 5 Th h l g f t ti g

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 25110

15 The psychology of testing

25

ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)

ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects

bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs

Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)

Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts

bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence

2 1 1 The V testing model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 26110

211 The V testing model

26

211 The V testing model ndash Verification amp Validationifi i C fi i b

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 11110

115 Why is testing necessary - How much testing is enough

when to stop testingThe five basic criteria often used to decide are

bull Previously defined coverage goals have been metbull The defect discovery rate has dropped below a previously defined thresholdbull The cost of finding the next defect exceeds the expected loss from that defectbull The project team reaches consensus that it is appropriate to release the productbull The manager decides to deliver the product

All these criteria are risk basedIt is important not depending on only one stopping criterion

Software Reliability Engineering can help also to determine when to stop testingby taking into consideration aspects like failure intensity

11

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 12110

12 What is testing - Defini tion of testing

12

Testing = the process concerned with planning the necessary static and dynamic

activities preparation and evaluation of software products and related deliverablesin order tobull determine that they satisfy specified requirementsbull demonstrate that they are fit for the intended usebull detect defects help and motivate the developers to fix thembull measure assess and improve the quality of the software product

Testing should be performed throughout the whole software life cycle

There are two basic types of testing execution and non-execution based

Other definitions

bull(IEEE) Testing = the process of analyzing a software item to detect the differencesbetween existing and required conditions and to evaluate its featuresbull(Myers (b3)) Testing = the process of executing a program with the intent of findingerrorsbull(Craig amp Jaskiel (b5)) Testing = a concurrent lifecycle process of engineering usingand maintaining test-ware in order to measure and improve the quality of thesoftware being tested

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 13110

12 What is testing ndash Testing ldquo schoolsrdquo

13

Analytic School - testing is rigorous academic and technicalbullTesting is a branch of CSMathematicsbullTesting techniques must have a logic-mathematical formbullKey Question Which techniques should we usebullRequire precise and detailed specifications

Factory School - testing is a way to measure progress with emphasis on cost and

repeatable standardsbullTesting must be managed amp cost effectivebullTesting validates the product amp measures development progressbullKey Questions How can we measure whether wersquore making progress When will we be donebullRequire clear boundaries between testing and other activities (startstop criteria)

bullEncourage standards (V-model) ldquobest practicesrdquo and certificationQuality School - emphasizes process amp quality acting as the gatekeeper bullSoftware quality requires disciplinebullTesters may need to police developers to follow the rulesbullKey Question Are we following a good process

bullTesting is a stepping stone to ldquoprocess improvementrdquoContext-Driven School - emphasizes people setting out to find the bugs that will bemost important to stakeholders

bullSoftware is created by people People set the contextbullTesting finds bugs acting as a skilled mental activitybullKey Question What testing would be most valuable right nowbullExpect changes Adapt testing plans based on test resultsbullTesting research requires empirical and psychological study

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 14110

13 General testing principles

14

bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects

bull Complete exhaustive testing is impossible good strategy and risk managementmust be used

bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs

bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)

bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped

bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)

bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 15110

13 General testing principles ndash heuristics of software testing

bullOperability - The better it works the more efficiently it can be tested

bullObservability - What we see is what we test

bull Controllability - The better we control the software the more the testing processcan be automated and optimized

bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing

bull Simplicity - The less there is to test the more quickly we can test it

bull Stability - The fewer the changes the fewer are the disruptions to testing

bull Understandability - The more information we will have the smarter we will test

bull Suitability - The more we know about the intended use of the software the

better we can organize our testing to find important bugs15

1 4 1 F d t l t t h

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 16110

141 Fundamental test process ndash phases

-Test Planning amp Test control

-Test Analysis amp Design

-Test Implementation ampExecution

-Evaluating exit criteria ampReporting

-Test Closure activities

16

1 4 1 Fundamental test process planning amp control

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 17110

141 Fundamental test process ndash planning amp control

Planning1 Determine scope

bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations

2 Determine risksbull Choose quality risk analysis method (eg FMEA)

bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule

bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies

4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria

Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions

17

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 18110

142 Fundamental test process ndash analysis amp design

bull Reviewing the test basis (such as requirements architecture designinterfaces)

bull Identifying test conditions or test requirements and required test data

based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-

conditionsbull Identify possible test Oracles

bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required

infrastructure and tools

(see Lee Copeland (b2))

18

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 19110

142 Fundamental test process ndash what is a test Oracle

19

bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be

executed The Test Oracle

Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative

hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead

lsquoproblem - no problemrsquo

Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles

Possible issues

bull false alarmsbull missed bugs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 20110

143 Fundamental test process ndash implementation amp execution

Test implementation

bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)

bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission

(see Rex Black (b4))

20

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 21110

143 Fundamental test process ndash priori tizing the Test Cases

Why prioritize the Test Cases

bullIt is not possible to test everything we must do our best in the time available

bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence

bullThis means we must prioritise and focus testing on the priorities

What to watch

bullSeverity of possible defectsbullProbability of possible defects

bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity

21

144 Fundamental test process ndash evaluating exit criteria and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 22110

144 Fundamental test process evaluating exit criteria andreporting

Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed

bull Check if testing mission should be changed

Test report ing

bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity

(see Rex Black (b4) amp RUP- Test discipline(s5))

22

1 4 5 F d l l

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 23110

145 Fundamental test process ndash test closure

bull Verify if test deliverables have been delivered

bull Check and close the remaining active bug reports

bull Archiving the test-ware and environment

bull Handover of the test environment

bull Analyze the identified test process problems (lessons learned)

bull Implement action plan based improvements

(see Rex Black (b4))

23

1 5 Th h l f i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 24110

15 The psychology of testing

24

Testing is regarded as a lsquodestructiversquo activity

(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts

bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise

bullShould be a planned organised kind ofperson

Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices

Rex BlackrsquosTop 10 professional errors

bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations

(see also Brian Marickrsquos article )

1 5 Th h l g f t ti g

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 25110

15 The psychology of testing

25

ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)

ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects

bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs

Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)

Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts

bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence

2 1 1 The V testing model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 26110

211 The V testing model

26

211 The V testing model ndash Verification amp Validationifi i C fi i b

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 12110

12 What is testing - Defini tion of testing

12

Testing = the process concerned with planning the necessary static and dynamic

activities preparation and evaluation of software products and related deliverablesin order tobull determine that they satisfy specified requirementsbull demonstrate that they are fit for the intended usebull detect defects help and motivate the developers to fix thembull measure assess and improve the quality of the software product

Testing should be performed throughout the whole software life cycle

There are two basic types of testing execution and non-execution based

Other definitions

bull(IEEE) Testing = the process of analyzing a software item to detect the differencesbetween existing and required conditions and to evaluate its featuresbull(Myers (b3)) Testing = the process of executing a program with the intent of findingerrorsbull(Craig amp Jaskiel (b5)) Testing = a concurrent lifecycle process of engineering usingand maintaining test-ware in order to measure and improve the quality of thesoftware being tested

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 13110

12 What is testing ndash Testing ldquo schoolsrdquo

13

Analytic School - testing is rigorous academic and technicalbullTesting is a branch of CSMathematicsbullTesting techniques must have a logic-mathematical formbullKey Question Which techniques should we usebullRequire precise and detailed specifications

Factory School - testing is a way to measure progress with emphasis on cost and

repeatable standardsbullTesting must be managed amp cost effectivebullTesting validates the product amp measures development progressbullKey Questions How can we measure whether wersquore making progress When will we be donebullRequire clear boundaries between testing and other activities (startstop criteria)

bullEncourage standards (V-model) ldquobest practicesrdquo and certificationQuality School - emphasizes process amp quality acting as the gatekeeper bullSoftware quality requires disciplinebullTesters may need to police developers to follow the rulesbullKey Question Are we following a good process

bullTesting is a stepping stone to ldquoprocess improvementrdquoContext-Driven School - emphasizes people setting out to find the bugs that will bemost important to stakeholders

bullSoftware is created by people People set the contextbullTesting finds bugs acting as a skilled mental activitybullKey Question What testing would be most valuable right nowbullExpect changes Adapt testing plans based on test resultsbullTesting research requires empirical and psychological study

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 14110

13 General testing principles

14

bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects

bull Complete exhaustive testing is impossible good strategy and risk managementmust be used

bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs

bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)

bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped

bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)

bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 15110

13 General testing principles ndash heuristics of software testing

bullOperability - The better it works the more efficiently it can be tested

bullObservability - What we see is what we test

bull Controllability - The better we control the software the more the testing processcan be automated and optimized

bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing

bull Simplicity - The less there is to test the more quickly we can test it

bull Stability - The fewer the changes the fewer are the disruptions to testing

bull Understandability - The more information we will have the smarter we will test

bull Suitability - The more we know about the intended use of the software the

better we can organize our testing to find important bugs15

1 4 1 F d t l t t h

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 16110

141 Fundamental test process ndash phases

-Test Planning amp Test control

-Test Analysis amp Design

-Test Implementation ampExecution

-Evaluating exit criteria ampReporting

-Test Closure activities

16

1 4 1 Fundamental test process planning amp control

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 17110

141 Fundamental test process ndash planning amp control

Planning1 Determine scope

bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations

2 Determine risksbull Choose quality risk analysis method (eg FMEA)

bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule

bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies

4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria

Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions

17

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 18110

142 Fundamental test process ndash analysis amp design

bull Reviewing the test basis (such as requirements architecture designinterfaces)

bull Identifying test conditions or test requirements and required test data

based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-

conditionsbull Identify possible test Oracles

bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required

infrastructure and tools

(see Lee Copeland (b2))

18

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 19110

142 Fundamental test process ndash what is a test Oracle

19

bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be

executed The Test Oracle

Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative

hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead

lsquoproblem - no problemrsquo

Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles

Possible issues

bull false alarmsbull missed bugs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 20110

143 Fundamental test process ndash implementation amp execution

Test implementation

bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)

bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission

(see Rex Black (b4))

20

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 21110

143 Fundamental test process ndash priori tizing the Test Cases

Why prioritize the Test Cases

bullIt is not possible to test everything we must do our best in the time available

bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence

bullThis means we must prioritise and focus testing on the priorities

What to watch

bullSeverity of possible defectsbullProbability of possible defects

bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity

21

144 Fundamental test process ndash evaluating exit criteria and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 22110

144 Fundamental test process evaluating exit criteria andreporting

Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed

bull Check if testing mission should be changed

Test report ing

bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity

(see Rex Black (b4) amp RUP- Test discipline(s5))

22

1 4 5 F d l l

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 23110

145 Fundamental test process ndash test closure

bull Verify if test deliverables have been delivered

bull Check and close the remaining active bug reports

bull Archiving the test-ware and environment

bull Handover of the test environment

bull Analyze the identified test process problems (lessons learned)

bull Implement action plan based improvements

(see Rex Black (b4))

23

1 5 Th h l f i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 24110

15 The psychology of testing

24

Testing is regarded as a lsquodestructiversquo activity

(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts

bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise

bullShould be a planned organised kind ofperson

Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices

Rex BlackrsquosTop 10 professional errors

bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations

(see also Brian Marickrsquos article )

1 5 Th h l g f t ti g

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 25110

15 The psychology of testing

25

ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)

ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects

bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs

Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)

Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts

bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence

2 1 1 The V testing model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 26110

211 The V testing model

26

211 The V testing model ndash Verification amp Validationifi i C fi i b

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 13110

12 What is testing ndash Testing ldquo schoolsrdquo

13

Analytic School - testing is rigorous academic and technicalbullTesting is a branch of CSMathematicsbullTesting techniques must have a logic-mathematical formbullKey Question Which techniques should we usebullRequire precise and detailed specifications

Factory School - testing is a way to measure progress with emphasis on cost and

repeatable standardsbullTesting must be managed amp cost effectivebullTesting validates the product amp measures development progressbullKey Questions How can we measure whether wersquore making progress When will we be donebullRequire clear boundaries between testing and other activities (startstop criteria)

bullEncourage standards (V-model) ldquobest practicesrdquo and certificationQuality School - emphasizes process amp quality acting as the gatekeeper bullSoftware quality requires disciplinebullTesters may need to police developers to follow the rulesbullKey Question Are we following a good process

bullTesting is a stepping stone to ldquoprocess improvementrdquoContext-Driven School - emphasizes people setting out to find the bugs that will bemost important to stakeholders

bullSoftware is created by people People set the contextbullTesting finds bugs acting as a skilled mental activitybullKey Question What testing would be most valuable right nowbullExpect changes Adapt testing plans based on test resultsbullTesting research requires empirical and psychological study

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 14110

13 General testing principles

14

bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects

bull Complete exhaustive testing is impossible good strategy and risk managementmust be used

bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs

bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)

bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped

bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)

bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 15110

13 General testing principles ndash heuristics of software testing

bullOperability - The better it works the more efficiently it can be tested

bullObservability - What we see is what we test

bull Controllability - The better we control the software the more the testing processcan be automated and optimized

bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing

bull Simplicity - The less there is to test the more quickly we can test it

bull Stability - The fewer the changes the fewer are the disruptions to testing

bull Understandability - The more information we will have the smarter we will test

bull Suitability - The more we know about the intended use of the software the

better we can organize our testing to find important bugs15

1 4 1 F d t l t t h

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 16110

141 Fundamental test process ndash phases

-Test Planning amp Test control

-Test Analysis amp Design

-Test Implementation ampExecution

-Evaluating exit criteria ampReporting

-Test Closure activities

16

1 4 1 Fundamental test process planning amp control

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 17110

141 Fundamental test process ndash planning amp control

Planning1 Determine scope

bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations

2 Determine risksbull Choose quality risk analysis method (eg FMEA)

bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule

bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies

4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria

Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions

17

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 18110

142 Fundamental test process ndash analysis amp design

bull Reviewing the test basis (such as requirements architecture designinterfaces)

bull Identifying test conditions or test requirements and required test data

based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-

conditionsbull Identify possible test Oracles

bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required

infrastructure and tools

(see Lee Copeland (b2))

18

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 19110

142 Fundamental test process ndash what is a test Oracle

19

bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be

executed The Test Oracle

Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative

hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead

lsquoproblem - no problemrsquo

Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles

Possible issues

bull false alarmsbull missed bugs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 20110

143 Fundamental test process ndash implementation amp execution

Test implementation

bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)

bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission

(see Rex Black (b4))

20

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 21110

143 Fundamental test process ndash priori tizing the Test Cases

Why prioritize the Test Cases

bullIt is not possible to test everything we must do our best in the time available

bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence

bullThis means we must prioritise and focus testing on the priorities

What to watch

bullSeverity of possible defectsbullProbability of possible defects

bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity

21

144 Fundamental test process ndash evaluating exit criteria and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 22110

144 Fundamental test process evaluating exit criteria andreporting

Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed

bull Check if testing mission should be changed

Test report ing

bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity

(see Rex Black (b4) amp RUP- Test discipline(s5))

22

1 4 5 F d l l

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 23110

145 Fundamental test process ndash test closure

bull Verify if test deliverables have been delivered

bull Check and close the remaining active bug reports

bull Archiving the test-ware and environment

bull Handover of the test environment

bull Analyze the identified test process problems (lessons learned)

bull Implement action plan based improvements

(see Rex Black (b4))

23

1 5 Th h l f i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 24110

15 The psychology of testing

24

Testing is regarded as a lsquodestructiversquo activity

(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts

bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise

bullShould be a planned organised kind ofperson

Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices

Rex BlackrsquosTop 10 professional errors

bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations

(see also Brian Marickrsquos article )

1 5 Th h l g f t ti g

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 25110

15 The psychology of testing

25

ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)

ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects

bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs

Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)

Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts

bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence

2 1 1 The V testing model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 26110

211 The V testing model

26

211 The V testing model ndash Verification amp Validationifi i C fi i b

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 14110

13 General testing principles

14

bull Testing shows presence of defects but cannot prove that there are no moredefects testing can only reduce the probability of undiscovered defects

bull Complete exhaustive testing is impossible good strategy and risk managementmust be used

bull Pareto rule (defect clustering) usually 20 of the modules contain 80 of thebugs

bull Early testing testing activities should start as soon as possible (including hereplanning design reviews)

bull Pesticide paradox if the same set of tests are repeated over again no new bugswill be found the test cases should be reviewed modified and new test casesdeveloped

bull Context dependence test design and execution is context dependent (desktopweb applications real-time hellip)

bull Verification and Validation discovering defects cannot help a product that is not fitto the users needs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 15110

13 General testing principles ndash heuristics of software testing

bullOperability - The better it works the more efficiently it can be tested

bullObservability - What we see is what we test

bull Controllability - The better we control the software the more the testing processcan be automated and optimized

bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing

bull Simplicity - The less there is to test the more quickly we can test it

bull Stability - The fewer the changes the fewer are the disruptions to testing

bull Understandability - The more information we will have the smarter we will test

bull Suitability - The more we know about the intended use of the software the

better we can organize our testing to find important bugs15

1 4 1 F d t l t t h

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 16110

141 Fundamental test process ndash phases

-Test Planning amp Test control

-Test Analysis amp Design

-Test Implementation ampExecution

-Evaluating exit criteria ampReporting

-Test Closure activities

16

1 4 1 Fundamental test process planning amp control

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 17110

141 Fundamental test process ndash planning amp control

Planning1 Determine scope

bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations

2 Determine risksbull Choose quality risk analysis method (eg FMEA)

bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule

bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies

4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria

Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions

17

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 18110

142 Fundamental test process ndash analysis amp design

bull Reviewing the test basis (such as requirements architecture designinterfaces)

bull Identifying test conditions or test requirements and required test data

based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-

conditionsbull Identify possible test Oracles

bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required

infrastructure and tools

(see Lee Copeland (b2))

18

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 19110

142 Fundamental test process ndash what is a test Oracle

19

bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be

executed The Test Oracle

Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative

hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead

lsquoproblem - no problemrsquo

Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles

Possible issues

bull false alarmsbull missed bugs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 20110

143 Fundamental test process ndash implementation amp execution

Test implementation

bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)

bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission

(see Rex Black (b4))

20

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 21110

143 Fundamental test process ndash priori tizing the Test Cases

Why prioritize the Test Cases

bullIt is not possible to test everything we must do our best in the time available

bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence

bullThis means we must prioritise and focus testing on the priorities

What to watch

bullSeverity of possible defectsbullProbability of possible defects

bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity

21

144 Fundamental test process ndash evaluating exit criteria and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 22110

144 Fundamental test process evaluating exit criteria andreporting

Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed

bull Check if testing mission should be changed

Test report ing

bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity

(see Rex Black (b4) amp RUP- Test discipline(s5))

22

1 4 5 F d l l

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 23110

145 Fundamental test process ndash test closure

bull Verify if test deliverables have been delivered

bull Check and close the remaining active bug reports

bull Archiving the test-ware and environment

bull Handover of the test environment

bull Analyze the identified test process problems (lessons learned)

bull Implement action plan based improvements

(see Rex Black (b4))

23

1 5 Th h l f i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 24110

15 The psychology of testing

24

Testing is regarded as a lsquodestructiversquo activity

(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts

bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise

bullShould be a planned organised kind ofperson

Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices

Rex BlackrsquosTop 10 professional errors

bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations

(see also Brian Marickrsquos article )

1 5 Th h l g f t ti g

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 25110

15 The psychology of testing

25

ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)

ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects

bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs

Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)

Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts

bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence

2 1 1 The V testing model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 26110

211 The V testing model

26

211 The V testing model ndash Verification amp Validationifi i C fi i b

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 15110

13 General testing principles ndash heuristics of software testing

bullOperability - The better it works the more efficiently it can be tested

bullObservability - What we see is what we test

bull Controllability - The better we control the software the more the testing processcan be automated and optimized

bull Decomposability - By controlling the scope of testing we can quickly isolateproblems and perform effective and efficient testing

bull Simplicity - The less there is to test the more quickly we can test it

bull Stability - The fewer the changes the fewer are the disruptions to testing

bull Understandability - The more information we will have the smarter we will test

bull Suitability - The more we know about the intended use of the software the

better we can organize our testing to find important bugs15

1 4 1 F d t l t t h

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 16110

141 Fundamental test process ndash phases

-Test Planning amp Test control

-Test Analysis amp Design

-Test Implementation ampExecution

-Evaluating exit criteria ampReporting

-Test Closure activities

16

1 4 1 Fundamental test process planning amp control

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 17110

141 Fundamental test process ndash planning amp control

Planning1 Determine scope

bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations

2 Determine risksbull Choose quality risk analysis method (eg FMEA)

bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule

bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies

4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria

Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions

17

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 18110

142 Fundamental test process ndash analysis amp design

bull Reviewing the test basis (such as requirements architecture designinterfaces)

bull Identifying test conditions or test requirements and required test data

based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-

conditionsbull Identify possible test Oracles

bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required

infrastructure and tools

(see Lee Copeland (b2))

18

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 19110

142 Fundamental test process ndash what is a test Oracle

19

bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be

executed The Test Oracle

Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative

hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead

lsquoproblem - no problemrsquo

Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles

Possible issues

bull false alarmsbull missed bugs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 20110

143 Fundamental test process ndash implementation amp execution

Test implementation

bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)

bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission

(see Rex Black (b4))

20

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 21110

143 Fundamental test process ndash priori tizing the Test Cases

Why prioritize the Test Cases

bullIt is not possible to test everything we must do our best in the time available

bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence

bullThis means we must prioritise and focus testing on the priorities

What to watch

bullSeverity of possible defectsbullProbability of possible defects

bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity

21

144 Fundamental test process ndash evaluating exit criteria and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 22110

144 Fundamental test process evaluating exit criteria andreporting

Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed

bull Check if testing mission should be changed

Test report ing

bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity

(see Rex Black (b4) amp RUP- Test discipline(s5))

22

1 4 5 F d l l

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 23110

145 Fundamental test process ndash test closure

bull Verify if test deliverables have been delivered

bull Check and close the remaining active bug reports

bull Archiving the test-ware and environment

bull Handover of the test environment

bull Analyze the identified test process problems (lessons learned)

bull Implement action plan based improvements

(see Rex Black (b4))

23

1 5 Th h l f i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 24110

15 The psychology of testing

24

Testing is regarded as a lsquodestructiversquo activity

(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts

bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise

bullShould be a planned organised kind ofperson

Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices

Rex BlackrsquosTop 10 professional errors

bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations

(see also Brian Marickrsquos article )

1 5 Th h l g f t ti g

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 25110

15 The psychology of testing

25

ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)

ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects

bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs

Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)

Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts

bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence

2 1 1 The V testing model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 26110

211 The V testing model

26

211 The V testing model ndash Verification amp Validationifi i C fi i b

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 16110

141 Fundamental test process ndash phases

-Test Planning amp Test control

-Test Analysis amp Design

-Test Implementation ampExecution

-Evaluating exit criteria ampReporting

-Test Closure activities

16

1 4 1 Fundamental test process planning amp control

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 17110

141 Fundamental test process ndash planning amp control

Planning1 Determine scope

bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations

2 Determine risksbull Choose quality risk analysis method (eg FMEA)

bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule

bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies

4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria

Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions

17

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 18110

142 Fundamental test process ndash analysis amp design

bull Reviewing the test basis (such as requirements architecture designinterfaces)

bull Identifying test conditions or test requirements and required test data

based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-

conditionsbull Identify possible test Oracles

bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required

infrastructure and tools

(see Lee Copeland (b2))

18

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 19110

142 Fundamental test process ndash what is a test Oracle

19

bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be

executed The Test Oracle

Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative

hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead

lsquoproblem - no problemrsquo

Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles

Possible issues

bull false alarmsbull missed bugs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 20110

143 Fundamental test process ndash implementation amp execution

Test implementation

bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)

bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission

(see Rex Black (b4))

20

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 21110

143 Fundamental test process ndash priori tizing the Test Cases

Why prioritize the Test Cases

bullIt is not possible to test everything we must do our best in the time available

bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence

bullThis means we must prioritise and focus testing on the priorities

What to watch

bullSeverity of possible defectsbullProbability of possible defects

bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity

21

144 Fundamental test process ndash evaluating exit criteria and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 22110

144 Fundamental test process evaluating exit criteria andreporting

Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed

bull Check if testing mission should be changed

Test report ing

bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity

(see Rex Black (b4) amp RUP- Test discipline(s5))

22

1 4 5 F d l l

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 23110

145 Fundamental test process ndash test closure

bull Verify if test deliverables have been delivered

bull Check and close the remaining active bug reports

bull Archiving the test-ware and environment

bull Handover of the test environment

bull Analyze the identified test process problems (lessons learned)

bull Implement action plan based improvements

(see Rex Black (b4))

23

1 5 Th h l f i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 24110

15 The psychology of testing

24

Testing is regarded as a lsquodestructiversquo activity

(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts

bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise

bullShould be a planned organised kind ofperson

Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices

Rex BlackrsquosTop 10 professional errors

bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations

(see also Brian Marickrsquos article )

1 5 Th h l g f t ti g

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 25110

15 The psychology of testing

25

ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)

ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects

bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs

Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)

Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts

bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence

2 1 1 The V testing model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 26110

211 The V testing model

26

211 The V testing model ndash Verification amp Validationifi i C fi i b

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 17110

141 Fundamental test process ndash planning amp control

Planning1 Determine scope

bull Study project documents used software life-cycle specifications product desiredquality attributesbull Clarify test process expectations

2 Determine risksbull Choose quality risk analysis method (eg FMEA)

bull Document the list of risks probability impact priority identify mitigation actions3 Estimate testing effort determine costs develop schedule

bull Define necessary rolesbull Decompose test project into phases and tasks (WBS)bull Schedule tasks assign resources set-up dependencies

4 Refine planbull Select test strategy (how to do it what test types at which test levels)bull Select metrics to be used for defect tracking coverage monitoringbull Define entry and exit criteria

Controlbull Measure and analyze resultsbull Monitor testing progress coverage exit criteriabull Assign or reallocate resources update the test plan schedulebull Initiate corrective actionsbull Make decisions

17

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 18110

142 Fundamental test process ndash analysis amp design

bull Reviewing the test basis (such as requirements architecture designinterfaces)

bull Identifying test conditions or test requirements and required test data

based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-

conditionsbull Identify possible test Oracles

bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required

infrastructure and tools

(see Lee Copeland (b2))

18

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 19110

142 Fundamental test process ndash what is a test Oracle

19

bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be

executed The Test Oracle

Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative

hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead

lsquoproblem - no problemrsquo

Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles

Possible issues

bull false alarmsbull missed bugs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 20110

143 Fundamental test process ndash implementation amp execution

Test implementation

bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)

bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission

(see Rex Black (b4))

20

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 21110

143 Fundamental test process ndash priori tizing the Test Cases

Why prioritize the Test Cases

bullIt is not possible to test everything we must do our best in the time available

bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence

bullThis means we must prioritise and focus testing on the priorities

What to watch

bullSeverity of possible defectsbullProbability of possible defects

bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity

21

144 Fundamental test process ndash evaluating exit criteria and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 22110

144 Fundamental test process evaluating exit criteria andreporting

Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed

bull Check if testing mission should be changed

Test report ing

bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity

(see Rex Black (b4) amp RUP- Test discipline(s5))

22

1 4 5 F d l l

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 23110

145 Fundamental test process ndash test closure

bull Verify if test deliverables have been delivered

bull Check and close the remaining active bug reports

bull Archiving the test-ware and environment

bull Handover of the test environment

bull Analyze the identified test process problems (lessons learned)

bull Implement action plan based improvements

(see Rex Black (b4))

23

1 5 Th h l f i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 24110

15 The psychology of testing

24

Testing is regarded as a lsquodestructiversquo activity

(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts

bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise

bullShould be a planned organised kind ofperson

Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices

Rex BlackrsquosTop 10 professional errors

bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations

(see also Brian Marickrsquos article )

1 5 Th h l g f t ti g

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 25110

15 The psychology of testing

25

ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)

ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects

bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs

Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)

Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts

bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence

2 1 1 The V testing model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 26110

211 The V testing model

26

211 The V testing model ndash Verification amp Validationifi i C fi i b

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 18110

142 Fundamental test process ndash analysis amp design

bull Reviewing the test basis (such as requirements architecture designinterfaces)

bull Identifying test conditions or test requirements and required test data

based on analysis of test items and the specificationbull Designing the testsbull Choose test techniquesbull Identify test scenarios pre-conditions expected results post-

conditionsbull Identify possible test Oracles

bull Evaluating testability of the requirements and systembull Designing the test environment set-up and identifying any required

infrastructure and tools

(see Lee Copeland (b2))

18

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 19110

142 Fundamental test process ndash what is a test Oracle

19

bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be

executed The Test Oracle

Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative

hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead

lsquoproblem - no problemrsquo

Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles

Possible issues

bull false alarmsbull missed bugs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 20110

143 Fundamental test process ndash implementation amp execution

Test implementation

bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)

bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission

(see Rex Black (b4))

20

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 21110

143 Fundamental test process ndash priori tizing the Test Cases

Why prioritize the Test Cases

bullIt is not possible to test everything we must do our best in the time available

bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence

bullThis means we must prioritise and focus testing on the priorities

What to watch

bullSeverity of possible defectsbullProbability of possible defects

bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity

21

144 Fundamental test process ndash evaluating exit criteria and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 22110

144 Fundamental test process evaluating exit criteria andreporting

Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed

bull Check if testing mission should be changed

Test report ing

bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity

(see Rex Black (b4) amp RUP- Test discipline(s5))

22

1 4 5 F d l l

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 23110

145 Fundamental test process ndash test closure

bull Verify if test deliverables have been delivered

bull Check and close the remaining active bug reports

bull Archiving the test-ware and environment

bull Handover of the test environment

bull Analyze the identified test process problems (lessons learned)

bull Implement action plan based improvements

(see Rex Black (b4))

23

1 5 Th h l f i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 24110

15 The psychology of testing

24

Testing is regarded as a lsquodestructiversquo activity

(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts

bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise

bullShould be a planned organised kind ofperson

Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices

Rex BlackrsquosTop 10 professional errors

bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations

(see also Brian Marickrsquos article )

1 5 Th h l g f t ti g

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 25110

15 The psychology of testing

25

ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)

ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects

bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs

Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)

Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts

bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence

2 1 1 The V testing model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 26110

211 The V testing model

26

211 The V testing model ndash Verification amp Validationifi i C fi i b

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 19110

142 Fundamental test process ndash what is a test Oracle

19

bull The expected result (test outcome) must be defined at the test analysis stagebull Who will decide that (expected result = actual result) when the test will be

executed The Test Oracle

Test Oracle = A source to determine expected result a principle or mechanism torecognize a problem The Test Oracle can bebull an existing system (the old versionhellip)bull a document (specification user manual)bull a competent client representative

hellip but never the source code itself Oracles in use = Simplification of Risk do not assess lsquopass - failrsquo but instead

lsquoproblem - no problemrsquo

Problem Oracles and Automation - Our ability to automate testing isfundamentally constrained by our ability to create and use oracles

Possible issues

bull false alarmsbull missed bugs

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 20110

143 Fundamental test process ndash implementation amp execution

Test implementation

bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)

bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission

(see Rex Black (b4))

20

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 21110

143 Fundamental test process ndash priori tizing the Test Cases

Why prioritize the Test Cases

bullIt is not possible to test everything we must do our best in the time available

bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence

bullThis means we must prioritise and focus testing on the priorities

What to watch

bullSeverity of possible defectsbullProbability of possible defects

bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity

21

144 Fundamental test process ndash evaluating exit criteria and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 22110

144 Fundamental test process evaluating exit criteria andreporting

Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed

bull Check if testing mission should be changed

Test report ing

bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity

(see Rex Black (b4) amp RUP- Test discipline(s5))

22

1 4 5 F d l l

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 23110

145 Fundamental test process ndash test closure

bull Verify if test deliverables have been delivered

bull Check and close the remaining active bug reports

bull Archiving the test-ware and environment

bull Handover of the test environment

bull Analyze the identified test process problems (lessons learned)

bull Implement action plan based improvements

(see Rex Black (b4))

23

1 5 Th h l f i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 24110

15 The psychology of testing

24

Testing is regarded as a lsquodestructiversquo activity

(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts

bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise

bullShould be a planned organised kind ofperson

Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices

Rex BlackrsquosTop 10 professional errors

bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations

(see also Brian Marickrsquos article )

1 5 Th h l g f t ti g

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 25110

15 The psychology of testing

25

ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)

ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects

bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs

Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)

Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts

bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence

2 1 1 The V testing model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 26110

211 The V testing model

26

211 The V testing model ndash Verification amp Validationifi i C fi i b

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 20110

143 Fundamental test process ndash implementation amp execution

Test implementation

bull Develop and prioritize test cases create test data test harnesses and automation scriptsbull Create test suites from the test casesbull Check test environmentTest executionbull Execute (manually or automatically) the test cases (suites)

bull Use Test Oracles to determine if test passed or failedbull Login the outcome of tests executionbull Report incidents (bugs) and try to discover if they are caused by the test data testprocedure or they are defect failuresbull Expand test activities as necessary according to the testing mission

(see Rex Black (b4))

20

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 21110

143 Fundamental test process ndash priori tizing the Test Cases

Why prioritize the Test Cases

bullIt is not possible to test everything we must do our best in the time available

bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence

bullThis means we must prioritise and focus testing on the priorities

What to watch

bullSeverity of possible defectsbullProbability of possible defects

bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity

21

144 Fundamental test process ndash evaluating exit criteria and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 22110

144 Fundamental test process evaluating exit criteria andreporting

Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed

bull Check if testing mission should be changed

Test report ing

bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity

(see Rex Black (b4) amp RUP- Test discipline(s5))

22

1 4 5 F d l l

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 23110

145 Fundamental test process ndash test closure

bull Verify if test deliverables have been delivered

bull Check and close the remaining active bug reports

bull Archiving the test-ware and environment

bull Handover of the test environment

bull Analyze the identified test process problems (lessons learned)

bull Implement action plan based improvements

(see Rex Black (b4))

23

1 5 Th h l f i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 24110

15 The psychology of testing

24

Testing is regarded as a lsquodestructiversquo activity

(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts

bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise

bullShould be a planned organised kind ofperson

Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices

Rex BlackrsquosTop 10 professional errors

bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations

(see also Brian Marickrsquos article )

1 5 Th h l g f t ti g

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 25110

15 The psychology of testing

25

ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)

ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects

bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs

Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)

Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts

bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence

2 1 1 The V testing model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 26110

211 The V testing model

26

211 The V testing model ndash Verification amp Validationifi i C fi i b

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 21110

143 Fundamental test process ndash priori tizing the Test Cases

Why prioritize the Test Cases

bullIt is not possible to test everything we must do our best in the time available

bullTesting must be Risk based assuring that the errors that will get through to theclientrsquos production system will have the smallest possible impact and frequencyof occurrence

bullThis means we must prioritise and focus testing on the priorities

What to watch

bullSeverity of possible defectsbullProbability of possible defects

bullVisibility of possible defectsbullClient Requirement importancebullBusiness or technical criticality of a featurebullFrequency of changes applied to a modulebullScenarios complexity

21

144 Fundamental test process ndash evaluating exit criteria and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 22110

144 Fundamental test process evaluating exit criteria andreporting

Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed

bull Check if testing mission should be changed

Test report ing

bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity

(see Rex Black (b4) amp RUP- Test discipline(s5))

22

1 4 5 F d l l

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 23110

145 Fundamental test process ndash test closure

bull Verify if test deliverables have been delivered

bull Check and close the remaining active bug reports

bull Archiving the test-ware and environment

bull Handover of the test environment

bull Analyze the identified test process problems (lessons learned)

bull Implement action plan based improvements

(see Rex Black (b4))

23

1 5 Th h l f i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 24110

15 The psychology of testing

24

Testing is regarded as a lsquodestructiversquo activity

(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts

bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise

bullShould be a planned organised kind ofperson

Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices

Rex BlackrsquosTop 10 professional errors

bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations

(see also Brian Marickrsquos article )

1 5 Th h l g f t ti g

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 25110

15 The psychology of testing

25

ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)

ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects

bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs

Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)

Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts

bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence

2 1 1 The V testing model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 26110

211 The V testing model

26

211 The V testing model ndash Verification amp Validationifi i C fi i b

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 22110

144 Fundamental test process evaluating exit criteria andreporting

Evaluate exit criteriabull Check test logs against exit criteria specified in test mission definitionbull Assess if more tests are needed

bull Check if testing mission should be changed

Test report ing

bull Write the test summary report for the stakeholders useThe test summary report should includebull Test Cases execution coverage ( executed )bull Test Cases Pass Fail bull Active bugs sorted according to their severity

(see Rex Black (b4) amp RUP- Test discipline(s5))

22

1 4 5 F d l l

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 23110

145 Fundamental test process ndash test closure

bull Verify if test deliverables have been delivered

bull Check and close the remaining active bug reports

bull Archiving the test-ware and environment

bull Handover of the test environment

bull Analyze the identified test process problems (lessons learned)

bull Implement action plan based improvements

(see Rex Black (b4))

23

1 5 Th h l f i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 24110

15 The psychology of testing

24

Testing is regarded as a lsquodestructiversquo activity

(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts

bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise

bullShould be a planned organised kind ofperson

Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices

Rex BlackrsquosTop 10 professional errors

bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations

(see also Brian Marickrsquos article )

1 5 Th h l g f t ti g

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 25110

15 The psychology of testing

25

ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)

ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects

bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs

Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)

Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts

bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence

2 1 1 The V testing model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 26110

211 The V testing model

26

211 The V testing model ndash Verification amp Validationifi i C fi i b

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 23110

145 Fundamental test process ndash test closure

bull Verify if test deliverables have been delivered

bull Check and close the remaining active bug reports

bull Archiving the test-ware and environment

bull Handover of the test environment

bull Analyze the identified test process problems (lessons learned)

bull Implement action plan based improvements

(see Rex Black (b4))

23

1 5 Th h l f i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 24110

15 The psychology of testing

24

Testing is regarded as a lsquodestructiversquo activity

(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts

bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise

bullShould be a planned organised kind ofperson

Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices

Rex BlackrsquosTop 10 professional errors

bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations

(see also Brian Marickrsquos article )

1 5 Th h l g f t ti g

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 25110

15 The psychology of testing

25

ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)

ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects

bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs

Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)

Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts

bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence

2 1 1 The V testing model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 26110

211 The V testing model

26

211 The V testing model ndash Verification amp Validationifi i C fi i b

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 24110

15 The psychology of testing

24

Testing is regarded as a lsquodestructiversquo activity

(we run tests to make the software failhellip) A good testerbullShould always have a critical approachbullMust keep attention to detailbullMust have analytical skillsbullShould have good verbal and writtencommunication skillsbullMust analyse incomplete facts

bullMust work with incomplete factsbullMust learn quickly about the product beingtestedbullShould be able to quickly prioritise

bullShould be a planned organised kind ofperson

Also he must have a good knowledge aboutbullThe customerrsquos business workflowsbullThe product architecture and interfacesbullThe software project processbullTesting techniques and practices

Rex BlackrsquosTop 10 professional errors

bullFall in Love with a ToolbullWrite Bad Bug ReportsbullFail to Define the MissionbullIgnore a Key StakeholderbullDeliver Bad News BadlybullTake Sole Responsibility for QualitybullBe an Un-appointed Process CopbullFail to Fire Someone who Needs FiringbullForget Yoursquore Providing a ServicebullIgnore Bad Expectations

(see also Brian Marickrsquos article )

1 5 Th h l g f t ti g

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 25110

15 The psychology of testing

25

ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)

ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects

bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs

Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)

Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts

bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence

2 1 1 The V testing model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 26110

211 The V testing model

26

211 The V testing model ndash Verification amp Validationifi i C fi i b

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 25110

15 The psychology of testing

25

ldquordquoThe best tester isnrsquot the one who finds the most bugs the best tester is the one whogets the most bugs fixedrdquo (Cem Kaner)

ldquo Sellingrdquo bugs (see Cem Kaner (c1)bull Motivate the programmer bull Demonstrate the bug effects

bull Overcome objectionsbull Increase the defect description coverage (indicate detailed preconditions behavior)bull Analyze the failurebull Produce a clear short unambiguous bug reportbull Advocate error costs

Levels of Independence of the Testing TeamLow ndash Developers write and execute their own testsMedium ndash Tests are written and executed by another developer High ndash Tests written and executed by an independent testing team (internal or external)

Testers Agile Manifesto (Jonathan Kohl)bullbug advocacy over bug counts

bulltestable software over exhaustive (requirements) docsbullmeasuring product success over measuring process successbullteam collaboration over departmental independence

2 1 1 The V testing model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 26110

211 The V testing model

26

211 The V testing model ndash Verification amp Validationifi i C fi i b

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 26110

211 The V testing model

26

211 The V testing model ndash Verification amp Validationifi i C fi i b

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 27110

27

Verification = Confirmation byexamination and through theprovision of objective evidence thatspecified requirements have beenfulfilled

Validation = Confirmation byexamination and through provision ofobjective evidence that therequirements for a specific intendeduse or application have been fulfilled

Verification is the dominant activity inthe Unit Integration System testinglevels Validation is a mandatoryactivity in the Acceptance testing level

2 1 1 The W testing model ndash dynamic testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 28110

211 The W testing model ndash dynamic testing

28

2 1 1 The W testing model ndash static testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 29110

211 The W testing model static testing

29

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 30110

212 Software development models Waterfall

30

212 Software development models - Waterfall

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 31110

212 Software development models Waterfall

Waterfall weaknesses

middot Linear any attempt to go back two or more phases to correct a problem ordeficiency results in major increases in cost and schedule

middot Integration problems usually surface too late Previously undetected errors ordesign deficiencies will emerge adding risk with little time to recover

middot Users cant see quality until the end They cant appreciate quality if thefinished product cant be seen

middot Deliverables are created for each phase and are considered frozen If thedeliverable of a phase changes which often happens the project will sufferschedule problemsThe entire software product is being worked on at one time There is no way to

partition the system for delivery of pieces of the system

31

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 32110

p p yp

32

212 Software development models - Rapid Prototype Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 33110

p p yp

Rapid Prototype Model weaknesses

middot In the rush to create a working prototype overall software quality orlong-term maintainability may be overlooked

middot Tendency for difficult problems to be pushed to the future causing theinitial promise of the prototype to not be met by subsequent products

middot Developers may fall into a code-and-fix cycle leading to expensiveunplanned prototype iterations

middot Customers become frustrated without the knowledge of the exactnumber of iterations that will be necessary

middot Users may have a tendency to add to the list of items to be prototypeduntil the scope of the project far exceeds the feasibility study

33

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 34110

34

212 Software development models - Incremental Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 35110

35

Incremental Model weaknesses

middot Definition of a complete fully functional system must be done early inthe life cycle to allow for the definition of the increments

middot The model does not allow for iterations within each incrementmiddot Because some modules will be completed long before others well-

defined interfaces are required

middot Requires good planning and design Management must take care todistribute the work the technical staff must watch dependencies

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 36110

36

212 Software development models - Spiral Model

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 37110

Spiral Model weaknesses

middot The model is complex and developers managers and customers may find ittoo complicated to usemiddot Considerable risk assessment expertise is requiredmiddot Hard to define objective verifiable milestones that indicate readiness to

proceed through the next iterationmiddot May be expensive - time spent planning resetting objectives doing risk

analysis and prototyping may be excessive

37

212 Software development models - Rational Unified Process

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 38110

38

213 Software development models ndash Testing life cycle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 39110

bull For each software activity there must be a corresponding testing activity

bull The objectives of the testing are specific to that ldquotestedrdquo activity

bull Plan analysis and design of a testing activity should be done during thecorresponding development activity

bull Review and inspection must be considered as part of testing activities

39

221 Test levels ndash Component testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 40110

40

bull Target single software modules components that are separately testable

bull Access to the code being tested is mandatory usually involves the programmer

bull May consist of

o Functional testso Non-functional tests (stress test)o Structural tests (statement coverage branch coverage)

bull Test cases follow the low level specification of the module

bull Can be automated (test driven software development)

o Develop first test codeo Then write the code to be testedo Execute until pass

bull Also named Unit testing

bull Good programming style (Design-by-contract respecting Demeterrsquos law)enhance the efficiency of Unit testing

222 Test levels ndash Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 41110

bull Target the interfaces between components and interfaces with other parts ofthe system

We focus on thedata

exchanged not on the tested functionalitiesbull Product software architecture understanding is critical

bull May consist ofo Functional testso Non-functional tests (ex performance test)o Component integration testing (after component testing)o System integration testing (after system testing)

bull Test strategy may be bottom-up top-down or big-bang

41

222 Test levels ndash Component Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 42110

Component integration testing (done after Component testing)

bullLinking a few components to check that they communicate correctly

bullIteratively linking more components together

bullVerify that data is exchanged between the components as required

bullIncrease the number of components create amp test subsystems and finally the

complete systemDrivers and Stubs should be used when necessary

bulldriver A software component or test tool that replaces a component that takescare of the control andor the calling of a component or system

bullstub A skeletal or special-purpose implementation of a software component usedto develop or test a component that calls or is otherwise dependent on it It replaces

a called component42

222 Test levels ndash System Integration testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 43110

System integration testing (done after System or Acceptance testing)

Testing the integration of systems and packages testing interfaces to externalorganizations

We check the data exchanged between our system and other external systems

Additional difficulties

bullMultiple PlatformsbullCommunications between platformsbullManagement of the environments

Approaches to access the external systems

bullTesting in a test environmentbullTesting in a lsquoclonersquo of a production environmentbullTesting in a real production environment

43

223 Test levels ndash System testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 44110

System testing = The process of testing an integrated system to verify that it meets

specified requirementsbull Target the whole product (system) as defined in the scope document

bull Environment issues are critical

bull May consist of

o Functional tests based on the requirement specificationso Non-functional tests (ex load tests)o Structural tests (ex web page links or menu item coverage)

bull Black box testing techniques may be used (ex business rule decision table)

bull Test strategy may be risk based

bull Test coverage is monitored

44

224 Test levels ndash Acceptance testingAcceptance Testing = Formal testing with respect to user needs requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 45110

Acceptance Testing = Formal testing with respect to user needs requirementsand business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user customers or other authorized entityto determine whether or not to accept the system

The main goalsbullestablish confidence in the systembullis the product good enough to be delivered to the client

The main focus is not to find defects but to assess the readiness for deploymentbullIt is not necessary the final testing level a final system integration testing sessioncan be executed after the acceptance testsbullMay be executed also after component testing (component usability acceptance)bullUsually involves client representatives

bullTypical formsbullUser acceptance business aware users verify the main featuresbullOperational acceptance testing backup-restore security maintenancebullAlpha and Beta testing performed by customers or potential users

bullAlpha at the developerrsquos sitebullBeta at the customerrsquos site

45

231 Test types ndash Functional testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 46110

Target Test the functionalities (features) of a product

bull Specification basedbull uses Test Cases derived from the specifications (Use Cases)bull business process based using business scenarios

bull Focused on checking the system against the specifications

bull Can be performed at all test levels

bull Considers the external behavior of the system

bull Black box design techniques will be used

bull

Security testing is part of functional testing related to the detection of threats

46

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 47110

232 Test types ndash Non-Functional testing - Usability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 48110

Usability testing = used to determine the extent towhich the software product is understood easy tolearn easy to operate and attractive to the usersunder specified conditions

bullPeople selected from the potential users maybe involved to study how they use the system

bullA quick and focused beta-test may be acheap way of doing Usability testing

bullThere is no simple way to examine howpeople will use the system

bullEasy to understood is not the same as easy tolearn or as easy to use or as easy to operate

48

232 Test types ndash Non-Functional testing - Instalability

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 49110

Instalability testing = The process of testing the installability of a softwareproductrdquo

bullDoes the installation work

bullHow easy is to install the system

bullDoes installation affect other software

bullDoes the environment affect the product

bullDoes it uninstall correctly

49

232 Test types ndash Non-Functional testing ndash Load StressPerformance Volume testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 50110

50

Load Test = A test type concerned with measuring the behavior of a componentor system with increasing load eg number of parallel users andor numbers oftransactions to determine what load can be handled by the component or system

Stress Test = Testing conducted to evaluate a system or component at orbeyond the limits of its specified requirements

Performance Test = The process of testing to determine the performance of asoftware product Performance can be measured watching

bullResponse timebullThroughputbullResources utilization

Spike Test = Keeping the system periodically for short amounts of timebeyond its specified limits

Endurance Test = a Load Test performed for a long time interval (week(s))

Volume Test = Testing where the system is subjected to large volumes of data

233 Test types ndash Structural testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 51110

bull Targeted to test

o internal structure (component)o architecture (system)

Uses only white box design techniquesbull Can be performed at all test levels

bull Used also to help measure the coverage ( of items being covered by tests)

bull Tool support is critical

51

234 Test types ndash Confirmation amp regression testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 52110

Confirmation testing = Re-testing of a module or product to confirm that thepreviously detected defect was fixed

bull Implies the use of a bug tracking toolbull Confirmation testing is not the same as the ldquodebuggingrdquo (debugging is adevelopment activity not a testing activity)

Regression testing = Re-testing of a previously tested program following

modification to ensure that defects have not been introduced or uncovered as aresult of the changes made It is performed when the software or its environmentis changed

bull

Can be performed at all test levelsbull Can be automated (because of cost and schedule reasons)

52

24 Maintenance testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 53110

Maintenance testing = Testing the changes to an operational system or the

impact of a changed environment to an operational system

Done on an existing operational system triggered by modification retirement ormigration of the software

Include

bullRelease based changesbullCorrective changesbullDatabase upgrades

Regression testing is also involved

Impact analysis = determining how the existing system may be affected bychanges (used to help decide how much regression testing to do)

53

31 Reviews and the testing process

S i T i i f ifi i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 54110

Static Testing = testing of a component or system at specification orimplementation level without execution of that software eg reviews (manual) orstatic code analysis (automated)

Reviews

Why reviewbullTo identify errors as soon as possible in the development lifecyclebullReviews offer the chance to find omissions and errors in the softwarespecifications

The target of a review is a software deliverablebull Specificationbull Use casebull Designbull Codebull Test casebull Manual

54

31 Reviews and the testing process

When to review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 55110

When to review

bullAs soon as an software artifact is produced before it is used as the basis for thenext step in development

Benefits includebull Early defect detectionbull Reduced testing costs and timebull Can find omissions

Risks bullIf misused they can lead to project team members frictionsbullThe errors amp omissions found should be regarded as a positive issuebullThe author should not take the errors amp omissions personallybullNo follow up to is made to ensure correction has been madebullWitch-hunts used when things are going wrong

55

321 Phases of a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 56110

Formal review phases

bullPlanning define scope select participants allocate roles define entry amp

exit criteriabullKick-off distribute documents explain objectives process check entrycriteriabullIndividual preparation each of participants studies the documents takesnotes issues questions and commentsbullReview meeting meeting participants discuss and log defects makerecommendationsbullRework fixing defects (by the author)bullFollow-up verify again gather metrics check exit criteria

56

322 Roles in a formal review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 57110

The formal reviews can use the following predefined roles

bull Manager schedules the review monitor entry and exit criteriabull Moderator distributes the documents leads the discussion mediates

various conflicting opinionsbull Author owner of the deliverable to be reviewedbull Reviewer technical domain experts identify and note findingsbull Scribe records and documents the discussions during the meeting

57

323 Types of review

Informal review Technical Review

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 58110

58

Informal reviewbullA peer or team lead reviews a softwaredeliverablebullWithout applying a formal processbullDocumentation of the review is optionalbullQuick way of finding omissions and defectsbullAmplitude and depth of the review dependson the reviewer bullMain purpose inexpensive way to getsome benefit

WalkthroughbullThe author of the deliverable leads thereview activity others participatebullPreparation of the reviewers is optionalbullScenario basedbullThe sessions are open-endedbullCan be informal but also formalbullMain purposes learning gainingunderstanding defect finding

Technical ReviewbullFormal Defect detection processbullMain meeting is preparedbullTeam includes peers and technical domainexpertsbullMay vary in practice from quite informal to veryformalbullLed by a moderator which is not the author bullChecklists may be used reports can be preparedbullMain purposes discuss make decisionsevaluate alternatives find defects solve technicalproblems and check conformance tospecifications and standards

InspectionFormal process based on checklists entry and

exit criteriaDedicated precise rolesLed by the moderator Metrics may be used in the assessmentReports list-of-findings are mandatory

Follow-up processMain purpose find defects

324 Success factors for reviews

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 59110

bull Clear objective is setbull Appropriate experts are involvedbull Identify issues not fix them on-the-spotbull Adequate psychological handling (author is not punished for the founddefects)bull

Level of formalism is adapted to the concrete situationbull Minimal preparation and trainingbull Management encourages learning process improvementbull Time-boxing is used to determine time allocated to each part of thedocument to be reviewedbull Use of effective checklists

59

33 Static analysis by tools

bullPerformed without executing the examined software but assisted by tools

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 60110

Performed without executing the examined software but assisted by tools

bullThe approach may be data flow or control flow basedbullBenefits

bull early defects detection

bull early warnings about unwanted code complexitybull detects missing links

bull improved maintainability of code and design

bull Typical defects discovered

bull reference to an un-initialized variable

bull never used variables

bull unreachable code

bull programming standards violations

bull security vulnerabilities

60

4 Test design techniques - glossary

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 61110

bull Test condition = item event attribute of a module or system thatcould be verified (ex feature structure element transaction qualityattribute)bull

Test data = data that affects or is affected by the execution of thespecific modulebull Test case [IEEE] = a set of input values execution preconditionsexpected results and execution post-conditions developed for a particular

objective or test condition such as to exercise a particular program path orto verify compliance with a specific requirementbull Test case specification [IEEE] = a document specifying a set of testcases for a test conditionbull Test procedure (suite) specification = a document specifying asequence of actions for the execution of a series of test cases

61

41 Test design ndash test development process

2 Develop test cases1 Identify test conditions

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 62110

62

middot Use cases are used as inputmiddot Test cases will cover all possible paths ofthe execution graph flowmiddot Test data should be specified if necessarymiddot Priorities of Test cases should be assigned

middot Traceability matrix (Use cases x Test cases)should be maintained

3 Develop test procedures

middot Group test cases into execution schedulesmiddot Factors to be considered

a Prioritizationb Logical dependenciesc Regression tests

middot Traceability from test condition to thespecifications (requirements) is a must

middot Risk analysis is a best practice

InputsbullField ndash levelbullGroup ndash level

Capability relatedbullTrigger conditionsbullConstraints or limitsbullInterfaces to other productsbullValidation of inputs at the followinglevels of aggregation

bullField action messagebullRecord row windowbullFile table screenbullDatabase

bullProduct statesbullBehavior rules

Architectural design relatedbullInvocation pathsbullCommunication pathsbullInternal data conditionsbullDesign statesbullExce tions

42 Test design ndash categories of test design techniques

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 63110

Black box no internal structure knowledge will be usedWhite box based on the analysis of the internal structureStatic without running exercised on specific project artifacts

Each black box or white box test technique has

bullA method (how to do it)bullA test case design approach (how to create test cases using the approach)bullA measurement technique ndash coverage (except the black box Syntax testing)

Other taxonomybull Specification based test cases are built from the specifications of the modulebull Structure based information about the module is constructed (design code) isused to derive the test casesbull

Experience based testerrsquos knowledge about the specific domain about thelikely defects is used

63

431 Black box techniques ndash equivalence partitioning

bull To minimize testing partition input (output) values into groups of equivalent

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 64110

64

g p p ( p ) g p qvalues (equivalent from the test outcome perspective)bull Select a value from each equivalence class as a representative value

If an input is a continuous range of values then there is typically one class of validvalues and two classes of invalid values one below the valid class and one

above it Example Rule for hiring a person is second its age

0 ndash 15 = do not hire

16 ndash 17 = part time18 ndash 54 = full time

55 -- 99 = do not hire

Which are the valid equivalence classes And the invalid ones

Give examples of representative values

(other examples)(see Lee Copeland b2 chap3 Cem Kaner c1 Paul Jorgensen b7 chap2263)

431 Black box techniques ndash all-pairs testing

In practice there are situations when a great number of combinations must be

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 65110

In practice there are situations when a great number of combinations must be

tested For example A Web site must operate correctly with different browsersmdash Internet Explorer 50 55 and 60 Netscape 60 61 and 70 Mozilla 11 andOpera 7 using different plug-insmdashRealPlayer MediaPlayer or none running ondifferent client operating systemsmdashWindows 95 98 ME NT 2000 and XPreceiving pages from different serversmdashIIS Apache and WebLogic running ondifferent server operating systemsmdashWindows NT 2000 and Linux

bullTest environment combinations

bull8 browsers

bull3 plug-ins

bull6 client operating systems

bull3 servers

bull3 server OS

bull1296 combinations

bull All-pairs testing is the solution tests a significant subset of variables pairs

65

432 Black box techniques ndash boundary value analysis

Boundaries = edges of the equivalence classes

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 66110

66

Boundaries edges of the equivalence classes

Boundary values = values at the edge and nearest to the edgeThe steps for using boundary values

bullFirst identify the equivalence classes

bullSecond identify the boundaries of each equivalence class

bullThird create test cases for each boundary value by choosing one point on theboundary one point just below the boundary and one point just above theboundary Below and above are relative terms and depend on the datavalues units

bullFor the previous example

bullboundary values are -101 14151615161716171817181954555698 99 100Or omitting duplicate values

-1011415161718195455569899100

(other examples)(see Lee Copeland b2 chap4 Paul Jorgensen b7 chap5)

433 Black box techniques ndash decision tables

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 67110

67

bull Conditions represent various input conditionsbull Actions are the actions taken depending on the various combinations of inputconditionsbull Each of the rules defines a unique combination of conditions that result in theexecution of the actions associated with that rulebull Actions do not depend on the condition evaluation order but only on theirvaluesbull Actions do not depend on any previous input conditions or system state

(see Lee Copeland b2 chap 5 Paul Jorgensen b7 chap 7)

433 Black box techniques ndash decision tables - example

a b c are they the edges of a triangle

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 68110

(however some additional test cases are needed)

68

434 Black box techniques ndash state transit ion tables

Allow the tester to interpret the system in term of

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 69110

p ybull Statesbull Transition between statesbull Events that trigger transitionsbull Actions resulting from the transitions

bull Transition table used

(see Lee Copeland b2 chap 7)

69

434 Black box techniques ndash state transition tables - example

Ticket buy - web application

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 70110

Exercise Fill-in the transition table

70

435 Black box techniques ndash requirements based testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 71110

Best practices

bullValidate requirements (what) against objectives (why)

bullApply use cases against requirements

bullPerform ambiguity reviews

bullInvolve domain experts in requirements reviews

bullCreate cause-effect diagrams

bullCheck logical consistency of test scenarios

bullValidate test scenarios with domain experts and users

bullWalk through scenarios comparing with design documents

bullWalk through scenarios comparing with code

71

435 Black box techniques ndash scenario testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 72110

Good scenario attributesbullIs based on a real story

bullIs motivating for the tester

bullIs crediblebullInvolves an enough complex use of environment and data

bullIs easy to evaluate ( no need for external oracle)

How to create good test scenariosbullWrite down real-life stories

bullList possible users analyze their interests and objectives

bullConsider also inexperienced or ostile usersbullList system benefits and create paths to access those features

bullWatch users using old versions of the system or an analog system

bullStudy complaints about other analog systems 72

435 Black box techniques ndash use case testing

Generating the Test Cases from the Use Cases

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 73110

Most common test case mistakes1 Making cases too long2 Incomplete incorrect or incoherent setup3 Leaving out a step4 Naming fields that changed or no longer exist5 Unclear whether tester or system does action6 Unclear what is a pass or fail result7 Failure to clean up

Steps1 Identify the use-case

scenarios2 For each scenario identify one

or more test cases3 For each test case identify theconditions that will cause it toexecute

4 Complete the test case byadding data values(see example)

73

435 Black box techniques ndash Syntax testing

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 74110

Syntax testing = uses a model of the formally-defined syntax of the inputs to aComponent

The syntax is represented as a number of rules each of which defines thepossible means of production of a symbol in terms of sequences of iterationsof or selections between other symbols

Here is a representation of the syntax for the floating point number float inBackus Naur Form (BNF)

float = int e intint = [+|-] natnat = digdig = 0|1|2|3|4|5|6|7|8|9

Syntax testing is the only black box technique without a coverage metricassigned

74

44 White box techniques ndash Control flow

Modules of code are converted to graphs the paths through the graphs are analyzed and

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 75110

g p p g g p y

test cases are created from that analysis There are different levels of coverage Process Blocks

A process block is a sequenceof program statements thatexecute sequentially

No entry into the block ispermitted except at thebeginning No exit from the

block is permitted except atthe end Once the block isinitiated every statementwithin it will be executedsequentially

75

Decision Point

A decision point is a point in themodule at which the controlflow can change Most decisionpoints are binary and areimplemented by if-then-elsestatements Multi-way decisionpoints are implemented bycase statements They arerepresented by a bubble withone entry and multiple exits

Junction Point

A junction point is a point atwhich control flows jointogether

Example

(see Lee Copeland b2 chap10)

441 White box techniques ndash statement coverage

Statement coverage = Executed statements Total executable statements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 76110

Example

a

if (b)

c

d

In case b is TRUE executing the code will result in 100 statement coverage

76

441 White box techniques ndash statement coverage - exercise

Given the code

a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 77110

if (x)

b

if (y)

c

else

d

else

e

x T T F F

y T F T F

a a a a

b b

c

d

e e

77How many test cases are needed to get 100 statement coverage

442 White box techniques ndash branch amp decision coverage - glossary

Branch = a conditional transfer of control from a statement to any other statement

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 78110

OR

= an unconditional transfer of control from a statement to any otherstatement except the next statement

Branch coverage = executed branches total branches

Decision coverage = executed decisions outcomes total decisions

For components with one entry point 100 Branch Coverage is equivalent to100 Decision Coverage

78

442 White box techniques ndash branch amp decision coverage - example

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 79110

Decisions = B2 B3 B5 each with 2 outcomes = 3 2 = 6Branches = (how many arrows) = 10

Q1 What are the decision and branch coverage for (B1 B2B9)

Q2 But for (B1-gtB2-gtB3-gtB4-gtB8-gtB2-gtB3-gtB5-gtB6-gtB8-gtB2-gtB3-gtB5-gtB7)

Answers 1 16 210 2 56 910

79

442 White box techniques ndash LCSAJ coverage

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 80110

LCSAJ = Linear Code Sequence and Jump

Defined by a triple conventionally identified by line numbers in a sourcecode listing

bullthe start of the linear code sequencebullthe end of the linear code sequence

bullthe target line to which control flow is transferred

LCSAJ coverage = executed LCSAJ sequences total nr of LCSAJ seq

80

443 White box techniques ndash data flow coverage

Just as one would not feel confident about a program without executing every

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 81110

statement in it as part of some test one should not feel confident about a programwithout having seen the effect of using the value produced by each and everycomputation

Data flow coverages

bullAll defs = Number of exercised definition-use pairs Number of variable definitions

bullAll c(omputation)-uses = Number of exercised definition- c-use pairs Number ofdefinition- c-use pairs

bullAll p(redicate)-uses = Number of exercised definition- p-use pairs Number ofdefinition- p-use pairs

bullAll uses = Number of exercised definition- use pairs Number of definition- use pairs

bullBranch condition = Boolean operand values executed Total Boolean operandvalues

bullBranch condition combination = Boolean operand values combinations executed Total Boolean operand values combinations

81(see Lee Copeland b2 chap11)

45 Exploratory testing

bull Exploratory testing = Concurrent test design test execution test logging

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 82110

and learning based on a quick test charter containing objectives and executedwithin delimited time-intervalsbull Uses structured approach to error guessing based on experienceavailable defect data domain expertisebull On-the-fly design of tests that attack these potential errorsbull Skill intuition and previous experience is vitalbull Test strategy is built around

bullThe project environment

bullQuality criteria defined for the project

bullElements of the product

bullRisk factors

82

46 Choosing test techniques

Factors used to choose

middot Product or system type middot Schedule constraints

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 83110

middot Standardsmiddot Productrsquos requirementsmiddot Available documentationmiddot Determined risks

middot Cost constraintsmiddot Used software development life cycle modelmiddot Testerrsquos skills and (domain) experience

(additional materials Unit Test design exercises )

83

511 Test organization amp independence

Options independent team or not

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 84110

Plusesbull Testers are not influenced by the other project membersbull Can act as lsquothe customerrsquos voicersquobull More objectivity in evaluating the product quality issues

Minusesbull Risk of isolation from the development teambull Communication issues

middot Developers can loose the lsquoquality ownershiprsquo attribute

84

512 Tasks of the test leader

bullPlan estimates test effort collaborates with project manager

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 85110

bullElaborates the test strategy

bullInitiate test specification implementation execution

bullSet-up configuration management of test environment amp deliverables

bullMonitors and controls the execution of tests

bullChooses suitable test metrics

bullDecides if and to what degree to automate the tests

bullSelect tools

bullSchedule tests

bullPrepare summary test reports

bullEvaluate test measurements

85

512 Tasks of the tester

Test Designer

bull

Test Analyst

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 86110

86

Define test approach (procedure)bull Structure test implementationbull Elaborates test case lists andwrites main test casesbull Assesses testability

Define testing environment details

Tester

bull Define test approach (procedure)bull

Write test casesbull Review test cases (peer review)bull Implement and execute tests

Record defects prepare defect reports

bull Identify test objectives (targets)bull Review product requirements andsoftware specificationsbull

Review test plans and test casesbull Verifies requirements to test casestraceabilitybull Define test scenario detailsbull Compares test results with testoraclebull Assesses test risksbull Gather test measures

521-522-523 ndash Test planning

Test plan = A document describing the scope approach resources and schedule of intendedtest activities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 87110

It identifies amongst others test items the features to be tested the testing tasks whowill do each task degree of tester independence the test environment the test designtechniques and test measurement techniques to be used and the rationale for theirchoice and any risks requiring contingency planning

It is a record of the test planning process

IEEE 829 Test plan contents

bullTest deliverablesbullTesting tasksbullEnvironmentbullResponsibilitiesbullStaffing and training needs

bullSchedulesbullRisks and contingenciesbullApprovals

bullTest plan identifier bullIntroductionbullTest itemsbullFeatures to be testedbullFeatures not to be tested

bullApproachbullItem pass fail criteriabullSuspension criteria amp resumptioncriteria

87

521-522-523 ndash Test planningbull Determine scope

o Study project documents usedsoftware life-cycle specifications product

bull Refine plano Define roles detailedresponsibilities

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 88110

88

desired quality attributeso Identify and communicate with otherstakeholderso Clarify test process expectations

bull

Determine riskso Choose quality risk analysis method(eg FMEA)o Document the list of risksprobability impact priority identify

mitigation actionsbull

Estimate testing effort determine costsdevelop schedule

o Define necessary roleso Decompose test project into phasesand tasks (WBS)o Schedule tasks assign resourcesset-up dependencieso Develop a budgeto Obtain commitment for the planfrom the stakeholders

bull

o Select test strategy test levelsTest strategy issues (alternatives)

bull Preventive approachbull

Reactive approachbull Risk-basedbull Model (standard) based

bull Choosing testing techniques (white

andor black box)o Select metrics to be used for defecttracking coverage monitoring

o Define entry and exit criteriabull Exit cr iteria

bull Coverage measuresbull Defect density or trend measuresbull Costbull Residual risk estimationbull Time or market based

524 ndash Test estimation

Two approaches

b d i (hi i l d )

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 89110

bull based on metrics (historical data)bull made by domain experts

Testing effort depends onbull product characteristics (complexity specification)

bull development process (team skills tools time factors)

bull defects discovered and rework involvedbull failure risk of the product (likelihood impact)

Time for confirmation testing and regression testing must beconsidered too

89

525 ndash Test strategiesTest approach ( test strategy ) = The chosen approaches and decisions made that follow from thetest projects and test teams goal or mission

The mission is typically effective and efficient testing and the strategies are the general policiesl d i i l h hi i i T i h ifi li i h i

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 90110

rules and principles that support this mission Test tactics are the specific policies techniquesprocesses and the way testing is done

One way to classify test approaches or strategies is based on the point in time at which the bulk ofthe test design work is begun

bullPreventative approaches where tests are designed as early as possible

bullReactive approaches where test design comes after the software or system has been produced

90

Or another taxonomy

Analytical - such as risk-basedtestingModel-based - such as stochastictestingMethodical - such as failure-

based experience-basedProcess- or standard-compliantDynamic and heuristic - such asexploratory testingConsultative

Regression-averse

53 ndash Test progress monitor ing reporting amp control

C t l id tif d i l t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 91110

Control identify and implementcorrective actions forbullTesting processbullOther software life-cycle activities

Possible corrective actions

bullAssign extra resource

bullRe-allocate resourcebullAdjust the test schedulebullArrange for extra testenvironmentsbullRefine the completion criteria

Monitoring - Test metrics used

bullTest cases ( passed failed)

bullDefects (found fixedfound densitytrends)

bullTest Coverage ( executed Test cases)

Reporting

bullDefects remaining

bullCoverage metrics

bullIdentified risks

91

54 ndash Configuration management

IEEE definition of Configuration Management A discipline applying technical and administrative direction and surveillance to

identif and doc ment the f nctional and ph sical characteristics of a

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 92110

92

bullidentify and document the functional and physical characteristics of aconfiguration itembullcontrol changes to those characteristicsbullrecord and report change processing and implementation status and

bullverify compliance with specified requirements

Configuration Managementbullidentifies the current configuration (hardware software) in the life cycle of

the system together with any changes that are in course of beingimplementedbullprovides traceability of changes through the lifecycle of the systembullpermits the reconstruction of a system whenever necessary

Only persistent objects must be subject to Configuration Managementtherefore the data processed by a system cannot be placed underConfiguration Management

bullRelated to Version Control and Change Control

54 ndash Configuration management

Configuration Management activities

Configuration identification = selecting the configuration items for a system

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 93110

Configuration identification = selecting the configuration items for a systemand recording their functional and physical characteristics in technicaldocumentation

Configuration control = evaluation co-ordination approval or disapproval andimplementation of changes to configuration items after formal establishment oftheir configuration identification

Status accounting = recording and reporting of information needed to managea configuration effectively includingbulla listing of the approved configuration identificationbullthe status of proposed changes to the configuration andbullthe implementation status of the approved changes

Configuration auditing = The function to check that the software productmatches the configuration items identified previously

93

54 ndash Configuration management

I T i C fi i M

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 94110

In Testing Configuration Management must

bullIdentify all test-ware items

bullEstablish and maintain the integrity of the testing deliverables(test plans test cases documentation) through the project lifecycle

bullSet and maintain the version of these itemsbullTrack the changes of these items

bullRelate test-ware items to other software development items inorder to maintain traceability

bullReference clearly all necessary documents in the test plansand test cases

94

55 ndash Risk amp Testing

Risk = a factor that could result in future negative consequencesexpressed as likelihood and impact

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 95110

bull Project risks (supplier related organizational technical)bull Product risks (defects delivered poor quality attributes (reliabilityusability performance)

The risks identified can be used tobull Define the test strategy and techniques to be usedbull Define the extent and depth of testingbull Prioritize test cases and procedures (find important defects early)bull Determine if review or training activities could help

95

56 ndash Incident management

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 96110

Incident = any significant unplanned event that occurs during testing thatrequires subsequent investigation and or correction

bullThe system does not function as expectedbullActual results differ from expected resultsbullRequired features are missing

Incident reports can be raised againstbulldocuments placed under review processbullproductrsquos defects related to functional amp non-functional requirementsbulldocumentation anomalies (manuals on-line help)bulltest-ware defects (errors in test cases or test procedures)

The incident reports raised against products defects are named also bugreports

96

56 ndash Incident management

Recommended Bug report format

bullDefect ID

Bug statuses

Issued ndash just been reported

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 97110

Defect IDbullComponent name and Build version

bullReported by and Date

bullError type

bullSeverity

bullPriority

bullSummary and detailed description

bullAttached documents

(Exercise )

Issued just been reportedOpened ndash programmer is working to solve-it

Fixed ndash programmer thinks thatrsquos repaired

Not solved ndash tester retested but the bug isnot solved

Deferred ndash programmer or PM decided topostpone the decision

Not-a-bug ndash programmer or testerdiscovered that it is not a defect

Closed ndash bug is solved and verified

97

611 ndash Test tool classification

bull Management of testing

bullTest management

bull Test execution

bullRecord and play

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 98110

Test managementbullRequirements management

bullBug tracking

bullConfiguration management

bull Static testing

bullReview support

bullStatic analysis

bullModeling

bull Test specification

bullTest design

bullTest data preparation

Record and playbullUnit test framework

bullResult comparators

bullCoverage measurement

bullSecurity

bull Performance and monitoring

bullDynamic analysis

bullLoad and stress testing

bullMonitoring

bull Specific application areas ( TTCN-3)

bull Other tools

98

612 ndash Tool support - Management of testing

bullTest management

bullManage testing activities

bullConfiguration management

bullIndividual support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 99110

Manage testing activitiesbullManage test-ware traceability

bullTest result reporting

bullTest metrics tools

bullRequirements management

bullChecking

bullTraceability

bullCoverage

bullBug tracking

Individual supportbullVersion and change control

bullBuilder

bullProject related

bullDepartment or company related

99

613 ndash Tool support - Static testing

bullReview support

bullProcess support

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 100110

bullCommunications support

bullTeam support

bullStatic analysis

bullCoding standards

bullWEB site structurebullMetrics

bullModeling bullSQL database management

100

614 ndash Tool support ndash Test specification

bullTest design

F i t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 101110

bullFrom requirements

bullFrom design models

bullTest stubs and driver generators

bullTest data preparation

101

615 ndash Tool support ndash Test execution and logging

bullRecord and play

bullScripting

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 102110

bullScripting

bullUnit test framework

bullTest harness frameworks

bullResult comparators

bullCoverage measurement

bullSecurity testing support

102

616 ndash Tool support ndash Performance and monitor ing

bullDynamic analysis

bullTime dependencies

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 103110

bullTime dependencies

bullMemory leaks

bullLoad testing

bullStress testing

bullMonitoring

Watch for possible mistakes

103

621 ndash Tool support ndash benefits

bullRepetitive work is reduced (eg running regression tests re-entering thesame test data and checking against coding standards)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 104110

bullGreater consistency and repeatability (eg tests executed by a tool andtests derived from requirements)

bullObjective assessment (eg static measures coverage and systembehavior)

bullEase of access to information about tests or testing (eg statistics andgraphs about test progress incident rates and performance)

104

621 ndash Tool support ndash risks

bullUnrealistic expectations for the tool (including functionality and ease ofuse)

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 105110

105

bullUnderestimating the time cost and effort for the initial introduction of a tool(including training and external expertise)

bullUnderestimating the time and effort needed to achieve significant andcontinuing benefits from the tool (including the need for changes in thetesting process and continuous improvement of the way the tool is used)

bullUnderestimating the effort required to maintain the test assets generated bythe toolbullOver-reliance on the tool (replacement for test design or where manualtesting would be better)

bullLack of a dedicated test automation specialistbullLack of good understanding and experience with the issues of testautomation

bullLack of stakeholders commitment for the implementation of a such tool

622 ndash Tool support ndash special considerations

bullTest execution tools

bullSignificant implementation effort

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 106110

Significant implementation effort

bullRecord and play tools are instable when changes occur

bullTechnical expertise is mandatory

bullPerformance testing tools

bullExpertise in design and results interpretation are mandatory

bullStatic analysis toolsbullLots of warnings generated

bullBuild management sensitive

bullTest management toolsbullInterfacing with other tools (Windows Office at least) is critical

bullTest tools future is much debated ( see hellip)

106

testing in Visual Studio Team System622 ndash Tool support ndash

bullDeveloper

bulluse Test DrivenDevelopment methods

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 107110

107

bullmanage Unit Testing

bullanalyze code coverage

bulluse code static analysisbulluse code profiler to handleperformance issues

bullTester

bullmanage test cases

bullmanage test suites

bullmanage manual testing

bullmanage bug trackingbullrecord play WEB tests

bullrun load tests

bullreport test results

622 ndash Tool support ndash testing in Agile distributed environment

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 108110

httpagile2008torontopbwikicomEvolution+of+tools+and+practices+of+a+distributed+agile+team 108

622 ndash Introducing a tool into an organization

bullTool selection process

bullIdentify requirements

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 109110

y q

bullIdentify constraints

bullCheck available tools on the market (feature evaluation)

bullEvaluate short list (feature comparison)

bullDemos

bullQuick pilotsbullSelect a tool

Note there are many free testing tools available some of them also online( wwwtestersdeskcom )

109

ISTQB Foundation Exam guidelines

K1 The candidates will recognize rememberand recall a term or concept

K2 The candidates can select the reasons orl ti f t t t l t d t th t i

bull 40 multiple (4) choice questionsbull 1 hour exam

bull Score gt= 65 (gt=26 good) t

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110

8112019 Curs Testare - Foundation

httpslidepdfcomreaderfullcurs-testare-foundation 110110

explanations for statements related to the topicThey can summarize compare classify andgive examples for concepts of testing

K3 The candidates can select the correctapplication of a concept or techniques andorapply it to a given context

answers) to passbull 50 K1 30 K2 20 K3

bullChapter 1 - 7 questions

bullChapter 2 ndash 6 questionsbullChapter 3 ndash 3 questionsbullChapter 4 ndash 12 questionsbullChapter 5 ndash 8 questions

bullChapter 6 ndash 4 questionsExample (see othershellip)Which statement regarding testing is correcta) Testing is planning specifying and executing a program with the aim of

finding defectsb) Testing is the process of correcting defects identified in a developedprogramc) Testing is to localize analyze and correct the direct defect cause

d) Testing is independently reviewing a system against its requirements 110