software testing. software testing is the execution of software with test data from the problem...

Software TestingSoftware Testing


Software testingSoftware testing is the execution is the execution of software with test data from the of software with test data from the problem domain.problem domain.

Static analysisStatic analysis (or static testing) (or static testing) involves analyzing the source code to involves analyzing the source code to determine errors.determine errors.

A A test casetest case is a specific input and is a specific input and the output that it should produce.the output that it should produce.


Exhaustive testingExhaustive testing is the execution is the execution of every possible test cases in the of every possible test cases in the problem domain (The set of all possible problem domain (The set of all possible inputs is called the inputs is called the domain spacedomain space).).

Generally a subset of the set of all Generally a subset of the set of all possible test cases is chosen.possible test cases is chosen.

The subset should be chosen so it is The subset should be chosen so it is representative of the set of all test representative of the set of all test cases.cases.


The two basic concerns in software The two basic concerns in software testing are:testing are:

1. What test cases to use (1. What test cases to use (test case test case selectionselection))

2. How many test cases are necessary 2. How many test cases are necessary ((stopping criterionstopping criterion).).

A test case should always include the A test case should always include the expected output.expected output.


Test case selection can be based on:Test case selection can be based on:

1. Specifications1. Specifications

2. The structure of the code2. The structure of the code

3. Data flow3. Data flow

4. Random selection4. Random selection

Test Coverage CriteriaTest Coverage Criteria

A A test coverage criteriatest coverage criteria is a rule is a rule about how to select test cases and about how to select test cases and when to stop testing.when to stop testing.

Test criterion A Test criterion A subsumessubsumes test test criterion B if any test that satisfies criterion B if any test that satisfies criterion A also satisfies criterion B.criterion A also satisfies criterion B.

Functional TestingFunctional Testing

In functional testing, the specification In functional testing, the specification of the software is used to identify of the software is used to identify subdomains that should be tested.subdomains that should be tested.

One way is to generate a test case One way is to generate a test case for every distinct ouput type.for every distinct ouput type.

Functional TestingFunctional Testing

Example: The lengths of three sides of a Example: The lengths of three sides of a triangle are entered and the type of triangle are entered and the type of triangle the lengths represent is output triangle the lengths represent is output (either scalene, isosceles or equilateral).(either scalene, isosceles or equilateral).

What is a “good” set of test cases?What is a “good” set of test cases?

Functional TestingFunctional Testing Scalene:Scalene:Increasing sizeIncreasing size 3, 4, 53, 4, 5Decreasing sizeDecreasing size 4, 3, 54, 3, 5Largest as secondLargest as second 4, 5, 34, 5, 3

IsoscelesIsoscelesa=b and c largera=b and c larger5, 5, 85, 5, 8a=c and b largera=c and b larger5, 8, 55, 8, 5b=c and a largerb=c and a larger8, 5, 58, 5, 5a=b and c smallera=b and c smaller 8, 8, 8, 8,

55a=c and b smallera=c and b smaller 8, 5, 8, 5,

88b=c and a smallerb=c and a smaller 5, 8, 5, 8,

88

Not a triangleNot a trianglelargest first largest first 6, 4, 6, 4,

22largest second largest second 4, 6, 4, 6,

22largest thirdlargest third 1, 2, 1, 2,

33 Bad InputsBad InputsOne badOne bad -1, 2, 4-1, 2, 4Two badTwo bad 3, -2, -53, -2, -5Three badThree bad 0, 0, 0, 0,

00

EquilateralEquilateralAll sides equalAll sides equal 5, 5, 5 5, 5, 5

Test MatricesTest Matrices

One way to formally identify One way to formally identify subdomains is to build a matrix with subdomains is to build a matrix with specifications (conditions) in the rows specifications (conditions) in the rows and subdomains in the columns and subdomains in the columns

Test MatricesTest MatricesConditions 1 2 3 4 5 6 7 8

a=b ora=c orb=c

T T T T T F F F

a=b andb=c

T T F F F F F F

a<=b+cOrb<=a+cOrc<=a+b

T F T T F T T F

a>=0 orb>=0 orc>=0

T F T F F T F F

Sample test case

0,0,0 3,3,3 0,4,0 3,8,3 5,8,5 0,5,6 3,4,8 3,4,5

Expected Output

Bad Inputs Equilateral Bad Inputs Nottriangle

Isosceles Bad Inputs Not triangle

Scalene

Structural TestingStructural Testing

Structural testing is based on the Structural testing is based on the structure of the source code.structure of the source code.

The criterion is that every statement The criterion is that every statement of source code should be executed of source code should be executed by some test case.by some test case.

The normal approach is to select test The normal approach is to select test cases until a coverage tool indicates cases until a coverage tool indicates that all statements in the code have that all statements in the code have been executed (Every statement been executed (Every statement coverage)coverage)

Structural TestingStructural TestingNode Source Line 3,4,5 3,5,3 0,1,0 4,4,4

A Read a, b, c * * * *

B Type=”scalene” * * * *

C if (a==b||b==c||a==c) * * * *

D Type=”isosceles” * * *

E if (a===b && b==c) * * * *

F Type=”equilateral” *

G If (a>=b+c ||b>=a+c||c>=a+b) * * * *

H Type=”not a triangle” *

I if (a<=0||b<=0||c<=0) * * * *

J Type=”bad inputs” *

K Print type * * * *


A more thorough test criterion is A more thorough test criterion is every branch coveringevery branch covering (called the (called the C1 test coverageC1 test coverage).).

In C1 test coverage, the goal is to go In C1 test coverage, the goal is to go each way out of every test decision.each way out of every test decision.


D

A,B,C

F

E

G

HI

JK

Control Flow Control Flow Graph for the Graph for the previous codeprevious code


Arcs 3,4,5 3,5,3 0,1,0 4,4,4

ABC-D * * *

ABC-E *

D-E * * *

E-F *

E-G * * *

F-G *

G-H *

G-I * *

H-I *

I-J *

I-K * * *

J-K *

Every Path TestingEvery Path Testing

A more thorough test is every path A more thorough test is every path testing which tests all possible paths testing which tests all possible paths through the control flow graph. through the control flow graph.

A path that cannot be taken is called A path that cannot be taken is called an infeasible path.an infeasible path.

Multiple Condition CoverageMultiple Condition Coverage

A A multiple condition testing criterionmultiple condition testing criterion requires that each primitive relation requires that each primitive relation condition is evaluated both true and false.condition is evaluated both true and false.

Lazy evaluation of expressions can Lazy evaluation of expressions can be used to eliminate casesbe used to eliminate cases

Multiple Condition CoverageMultiple Condition Coverage

if (a==b||b==c||a==c)if (a==b||b==c||a==c)

Combinations Possible Test Cases

Branch

TXX 3,3,4 ABC-D

FTX 4,3,3 ABC-D

FFT 3,4,3 ABC-D

FFF 3,4,5 ABC-E

Test PlansA test plan states: What the items to be tested areAt what level they will be tested What sequence they are to be tested inHow the test strategy will be applied to

the testing of each itemA description of the test environment.

What to consider for the Test Plan

From IEEE 829 Standard for Software Test Documentation:

1. Test Plan Identifier 2. References 3. Introduction 4. Test Items


5. Software Risk Issues 6. Features to be Tested 7. Features not to be Tested 8. Approach 9. Item Pass/Fail Criteria 10. Suspension Criteria and Resumption Requirements


11. Test Deliverables 12. Remaining Test Tasks 13. Environmental Needs 14. Staffing and Training Needs 15. Responsibilities

16. Schedule 17. Planning Risks and Contingencies 18. Approvals 19. Glossary

What I want included in the test plan

List of responsibilities (and who does them)– Who runs the test– Who approves the results

Unit tests to be run with expected results– Objectives– Approach– Include a description of the type of data used for

unit testing and why that data is being used– Pass/fail criteria– When the tests are to be run (timeline)


Integration tests to be run with expected results– Objectives– Approach– Include a description of the type of data

used for integration testing and why that data is being used

– Pass/fail criteria– When the tests are to be run (timeline)


Systems/Acceptance tests to be run with expected results– Objectives– Approach– Include a description of the type of data

used for systems/acceptance testing and why that data is being used

– Pass/fail criteria– When the tests are to be run (timeline)


Alpha testing to be done– Objectives– Approach– Features to be tested– Pass/fail criteria– When the tests are to be run (timeline)


Beta testing– Objectives– Approach– Who is participating in the beta testing– Pass/fail criteria– What I want included in the test plan

DeliverablesTest Design Specification:

detailing test conditions and the expected results as well as test pass criteria.

Test Case Specification: specifying the test data for use in running the test conditions identified in the Test Design Specification

DeliverablesTest Procedure Specification:

detailing how to run each test, including any set-up preconditions and the steps that need to be followed

Test Item Transmittal Report: reporting on when tested software components have progressed from one stage of testing to the next

DeliverablesTest Log: recording which tests

cases were run, who ran them, in what order, and whether each test passed or failed

DeliverablesTest Incident Report: detailing, for any

test that failed, the actual versus expected result, and other information intended to throw light on why a test has failed.

These include the expected results being wrong, the test being run wrongly, or inconsistency in the requirements meaning that more than one interpretation could be made.

Deliverables- Test Incident Report

The report consists of all details of the incident such as actual and expected results, when it failed, and any supporting evidence that will help in its resolution.

The report will also include, if possible, an assessment of the impact of an incident upon testing.

Deliverables- Test Summary Report

Test Summary Report: A management report providing any important information uncovered by the tests accomplished, and including:

Assessments of the quality of the testing effort

The quality of the software system under test, and statistics derived from Incident Reports.

Deliverables- Test Summary Report

The report also records:What testing was doneHow long it took in order to improve any

future test planning.

This final document is used to indicate whether the software system under test is fit for purpose according to whether or not it has met acceptance criteria defined by project stakeholders

IEEE Standards829-1983 IEEE Standard for Software

Test Documentation 1008-1987 IEEE Standard for

Software Unit Testing 1012-1986 IEEE Standard for Software Verification & Validation Plans

1059-1993 IEEE Guide for Software Verification & Validation Plans

Sample Test Planshttp://it.toolbox.com/blogs

/enterprise-solutions/testing-a-sample-test-plan-3283

software testing. software testing is the execution of software with test data from the problem...

Documents