Software Testing
2CMSC 345, Version 4/12
Topics The testing process
unit testing integration and system testing acceptance testing
Test case planning and tracking Testing types
black box vs. white box testing coverage testing regression testing stress testing
Test case prioritization
3CMSC 345, Version 4/12
Testing
Goal is to discover defects The only validation technique for non-
functional requirements Successful test: causes a program to
behave in an anomalous way Testing shows the presence, not the
absence, of defects.
4CMSC 345, Version 4/12
Unit Testing
Integration and System Testing
Acceptance Testing
Testing Process Overview
unit1
unit2
unit3
unitn
unitunit
unit
Subsystem1
unitunit
Subsystem2
unit
unitunit
unit
Subsystemm
Final System Final System
5CMSC 345, Version 4/12
unitunit
Subsystem
The Testing Process Elaborated Unit (component) testing
Testing of individual program components (e.g., functions, methods, or classes)
Usually the responsibility of the component’s developer Must be structured White box testing
Integration and system testing Testing of groups of components integrated to create a subsystem or
the entire system Sometimes the responsibility of an independent testing team Tests are based on system specification (black box testing) Is iterative (keeps building) May include alpha and beta testing
Acceptance testing Run in the presence of customer or by customer Used to validate all system requirements
unit
Final System$$$$!
6CMSC 345, Version 4/12
Unit (Component) Testing
You may do this on your own, but always … have a plan keep records automate!
JUnit (for Java) or other toolseven #ifdef - #endif pairs (C/C++) main( ) method (Java)
7CMSC 345, Version 4/12
Integration Testing Approaches – Procedural Programming
Top-down testing Start with high-level components and integrate from top down Uses stubs
Bottom-up testing Start with low-level components and integrate from bottom up Uses drivers
Sandwich testing Combination of top-down and bottom-up
Thread testing Tests one piece of functionality at a time
8CMSC 345, Version 4/12
n
Level
0
1
Top-down Integration & Testing
Being Tested
Stub StubStub
Stub: 1) says “I’ve been called!”, 2) confirms receipt of inputs, if any, and 3) returns hard-coded outputs, if necessary
9CMSC 345, Version 4/12
n
Level
0
1
Top-down Integration & Testing
Tested
Being Tested
Being Tested
Being Tested
Stub Stub StubStub
Stub: 1) says “I’ve been called!”, 2) confirms receipt of inputs, if any, and 3) returns hard-coded outputs, if necessary
10CMSC 345, Version 4/12
n
Level
0
1
Top-down Integration & Testing
Tested
Tested TestedTested
Tested TestedTestedTested
Being Tested
Being Tested
Being Tested
Being Tested
Stub: 1) says “I’ve been called!”, 2) confirms receipt of inputs, if any, and 3) returns hard-coded outputs, if necessary
11CMSC 345, Version 4/12
n
Level
0
1
Bottom-up Integration & Testing
Being Tested
Being Tested
Being Tested
Being Tested
Driver Driver
Driver: 1) Calls functions, 2) passes in inputs, if any, and 3) confirms receipt of outputs, if any
12CMSC 345, Version 4/12
n
Level
0
1
Bottom-up Integration & Testing
Being Tested
Being Tested
Being Tested
Being Tested
Tested TestedTestedTested
Driver: 1) Calls functions, 2) passes in inputs, if any, and 3) confirms receipt of outputs, if any
13CMSC 345, Version 4/12
n
Level
0
1
Bottom-up Integration & Testing
Tested TestedTestedTested
Tested TestedTestedTested
Driver DriverDriver
Driver: 1) Calls functions, 2) passes in inputs, if any, and 3) confirms receipt of outputs, if any
14CMSC 345, Version 4/12
n
Level
0
1
Sandwich Integration & TestingUses both stubs and drivers
15CMSC 345, Version 4/12
n
Level
0
1
Thread Integration & TestingMay be done top-down, bottom-up, or sandwich
16CMSC 345, Version 4/12
Exercise (on your own!)What are some of the benefits associated with
each of the following integration and testing methods?
Top-down
Bottom-up
Sandwich
Thread
17CMSC 345, Version 4/12
Integration Testing Approaches – Object-oriented Programming
Use based Start with classes that have no dependencies Move “out” to next “level” of classes, and so on Similar to bottom-up for procedural programming
Cluster Test clusters of associated classes Integrate the clusters May be combined with use based testing (above)
Scenario Based on functional specifications (possibly expressed as use
cases) Similar to thread testing for procedural programming
18CMSC 345, Version 4/12
System Testing Alpha
Internal to development organization, but not the developers themselves
BetaExternal to development organization
AcceptancePerformed in the presence of or possibly by
the customerDetermines final “acceptance” by the
customer
19CMSC 345, Version 4/12
Test #
Description
Input Values
Expected Output
Actual Output
P/F Criteria
Comments
Test Case Planning and Tracking
Natural language description
IDSpecific values (if possible)
Specific values (if possible)
Specific values (if possible)
Pass, Fail, other notes
How to determine Pass or Fail
20CMSC 345, Version 4/12
Test #
Description
Input Values
Expected Output
Actual Output
P/F Criteria
Comments
1 Tests that a string will be displayed centered and in 12 point
font = 12 alignment = “C” string = “CMSC 345”
“CMSC 345” is displayed and centered in 12 point
Actual = Expected
A Sample Testing Form Before Testing
21CMSC 345, Version 4/12
Test #
Description
Input Values
Expected Output
Actual Output
P/F Criteria
Comments
1 Tests that a string will be displayed centered and in 12 point
font = 12 alignment = “C” string = “CMSC 345”
“CMSC 345” is displayed and centered in 12 point
“CMSC 345 is displayed and centered in 12 point
Actual = Expected
Pass
A Sample Testing Form After Testing
22CMSC 345, Version 4/12
Test #
Description
Input Values
Expected Output
Actual Output
P/F Criteria
Comments
1 Tests that a string will be displayed centered and in 12 point
font = 12 alignment = “C” string = “CMSC 345”
“CMSC 345” is displayed and centered in 12 point
“CMSC 345” is displayed and centered in 12 point
Actual = Expected
Pass
2 Tests that the largest integer in the list is located
intList = 5, -2,100,16, -51,12 listSize = 6
100 Actual = Expected
A Sample Testing Form Before Testing
23CMSC 345, Version 4/12
Test #
Description
Input Values
Expected Output
Actual Output
P/F Criteria
Comments
1 Tests that a string will be displayed centered and in 12 point
font = 12 alignment = “C” string = “CMSC 345”
“CMSC 345” is displayed and centered in 12 point
“CMSC 345 is displayed and centered in 12 point
Actual = Expected
Pass
2 Tests that the largest integer in the list is located
intList = 5, -2,100,16, -51,12 listSize = 6
100 -51 Actual = Expected
Fail
A Sample Testing Form After Testing
24CMSC 345, Version 4/12
White Box Testing
Sometimes called structural testing Can “see inside” the code unit Objectives:
Exercise unique algorithmsExercise all program statements at least once
Typically tests small program units such as functions or class methods
code code code code code code code code code codecode code code code codecode code code code code
input output
25CMSC 345, Version 4/12
Coverage Testing A technique for white box testing Ensures that each statement is executed at least
once Use a program flow graph to derive paths Find minimum paths needed to “cover” all
statements (there are tools to do this) Drive the code through all of these paths All branches are executed, but not all
combinations of branches. Some paths may be impossible to test.
#include <iostream>using namespace std;void triangle (int, int, int); // parameters are sides of the triangle
int main(){ int s1, s2, s3; char repeat;
cout << "This program will accept the sides of a triangle\n" << " and tell you the type of triangle.\n" << endl; do {
cout << "Enter the 3 sides: " ;cin >> s1 >> s2 >> s3;triangle(s1, s2, s3);cout << "Want to do another? (Y or N) ";cin >> repeat;
cout << endl << endl; } while (repeat =='Y' || repeat == 'y'); cout <<"Program terminated" << endl << endl; cout << "Enter any character to continue" << endl; cin >> repeat; return 0;}
void triangle (int side1, int side2, int side3){ // Error checking/***********************************/ // Check sides for validity if (side1 <= 0 || side2 <= 0) // 1 cout << "Illegal value for side" << endl << endl; // 2 // Check Triangle inequality if (side1+side2 <= side3 || side2+side3 <= side1) // 3 cout << "Triangle inequality violation" << endl << endl; // 4/***********************************/ // Pick triangle type if (side1 == side2 && side2 == side3) // 5 cout << "Triangle is equilateral" << endl; // 6 else if (side1 == side2 || side2 == side3 || side1 == side3) // 7 cout << "Triangle is isosceles" << endl; // 8 else if (side1*side1 + side2*side2 == side3*side3 || // 9 side1*side1 + side3*side3 == side2*side2) cout << "Triangle is right" << endl; //10 else cout <<"Triangle is scalene" << endl; //11return; //12} D. Cheslock, Northrup Grumman, 2009
triangle.cpp
CMSC 345, Version 4/12 26
triangle.exe Control Graph
Vertices= 18Edges = 28Regions = 12
V(g) = 28 - 18 + 2 (e-n+2) = 11 + 1 (p+1) = 12
1
7c
7b
5b
1b
3b
1a
2
3a
4
5a
67a
8
9a
10
1112
9b
2
3
4
5
6
7 8
9
11
10
12
Baseline path
D. Cheslock, Northrup Grumman, 2009CMSC 345, Version 4/12 27
A Set Of Basis Paths For triangle.cpp 1) 1a - 1b - 3a - 3b - 5a - 7a - 7b -7c -9a - 9b - 11 - 12
sample data: 3, 5, 7 (baseline path)
2) 1a - 2 - 3a - 3b - 5a - 7a - 7b -7c -9a - 9b - 11 - 12
sample data: -3, 7, 3
3) 1a - 1b - 2 - 3a - 3b - 5a - 7a - 7b -7c - 9a - 9b - 11 - 12
sample data: can’t perform this test
4) 1a - 1b - 3a - 4 - 5a - 7a - 7b -7c - 9a - 9b - 11 - 12
sample data: 2, 3, 7
5) 1a - 1b - 3a - 3b - 4 - 5a - 7a - 7b -7c - 9a - 9b - 11 - 12
sample data: 7, 3, 2
6) 1a - 1b - 3a - 3b - 5a - 5b - 6 - 12
sample data: 5, 5, 5
7) 1a - 1b - 3a - 3b - 5a - 5b - 7a - 7b -7c - 9a - 9b - 11 - 12
sample data: can’t perform this test
8) 1a - 1b - 3a - 3b - 5a - 5b - 7a - 8 - 12
sample data: 4, 4, 2
9) 1a - 1b - 3a - 3b - 5a - 7a - 7b - 8 - 12
sample data: 2, 4, 4
10) 1a - 1b - 3a - 3b - 5a - 7a - 7b -7c - 8 - 12
sample data: 4, 2, 4
11) 1a - 1b - 3a - 3b - 5a - 7a - 7b -7c - 9a - 10 - 12
sample data: 3, 4, 5
12) 1a - 1b - 3a - 3b - 5a - 7a - 7b -7c - 9a - 9b - 10 - 12
sample data: 3, 5, 4 D. Cheslock, Northrup Grumman, 2009CMSC 345, Version 4/12 28
29CMSC 345, Version 4/12
Metrics – Cyclomatic Complexity
The minimum number of tests needed to execute all statements
CC = number_edges – number_nodes + 2
OR
CC = number_decisions + 1
OR
the number of regions
30CMSC 345, Version 4/12
Black Box Testing
Cannot “see inside” code unit Test cases are:
based on the system specification and/or interfaces (preconditions, postconditions, invariants, and parameters)
Use equivalence partitions when conducting black box testing
input output
31CMSC 345, Version 4/12
Equivalence Partitions and Boundary Testing
Equivalence Partition A group of input values that will produce
equivalent behavior Example
Square root function the set of all numbers >= zero the set of all negative numbers
Test cases should be chosen from each partition, especially at the boundaries.
32CMSC 345, Version 4/12
Equivalence Partitions and Boundary Testing
Exercise A system accepts 4 to 10 inputs that are 5-
digit integers >= 10,000 Partition the system inputs into groups
(partitions) that should cause equivalent behavior. Include both valid and invalid inputs.
Ian Somerville, Software Engineering, 6th edition
33CMSC 345, Version 4/12
Between 10000 and 99999Less than 10000 More than 99999
999910000 50000
10000099999
Input values
Between 4 and 10Less than 4 More than 10
34 7
1110
Number of input values
Ian Somerville, Software Engineering, 6th edition
The six partitions with corresponding boundary values
34CMSC 345, Version 4/12
Regression Testing
Code A
Test 1
Test 2
Code A
Code B
Code C
Test 1
Test 2
Test 3
Test 4
Test 5
Code A
Code B
Code C
Code D
Test 2
Test 3
Test 4
Test 5
Test 1
Test 6
Test 7
Test 8
… and so on
35CMSC 345, Version 4/12
Stress Testing Example: A child’s car seat Exercises the system beyond its maximum
design load. Examples: Exceed string lengths Store/manipulate more data than in specification Load system with more users than in specification
Often causes defects to come to light Systems should not fail catastrophically.
Stress testing checks for unacceptable loss of service or data.
36CMSC 345, Version 4/12
Test Case PrioritizationWhy? To locate defects quickly to:
begin debugging ASAP debug and deliver in increments ASAP
To locate the majority of errors quickly Other
How? Can prioritize according to:
module or functional dependencies module size module complexity module suspected to be the most error-prone most “important” module or functionality other
37CMSC 345, Version 4/12
References
Somerville, I., Software Engineering, 6th ed SourceForge.net, JUnit,
http://junit.sourceforge.net, accessed 11/6/07
JUnit.org, http://www.junit.org, accessed 11/6/07