- Abhishek Rakshe

- Rajasa Savant

Testing & Analysis of Web ApplicationsUniversity of Southern California (USC)Under the Guidance of Professor William G.J. Halfond

Outline

Introduction to Test Case Generator

Design & Approach

Algorithm

Implementation

Results

Analysis

Conclusion

Future Scope

Introduction to Test Case Generator Test Case Generator generates test suite of test

cases with desired line & branch coverage.

The tool automatically executes the test cases using Selenium TestNG and generates XML reports using another tool Cobertura.

The test cases chosen by the tool have highest line coverage and branch coverage.

The tool uses approximation technique to minimize the search space for selecting optimum test cases.

Design & Approach Test Case Generator makes use of the following tools:

Selenium 2.2.5

Selenium Standalone Server 2.2.5

TestNG

Cobertura 1.9.4.1

Eclipse IDE

Execute Test Cases

• Using Selenium TestNG automate test case execution in java

Generate XML reports

• Using Cobertura to generate reports

Get application meta data

• Parse the XML report and build application schema

Parse all test cases

• Using SAX xml parser in java. Build Test Case Suite Map

Optimize the test case data

• Get the line and branch overlap weightage

• Build Overlap Coverage Hash Map

• Get the chosen test cases on basis of the Overlap Coverage, line and branch coverage constraints

Algorithm – Optimize test case data 1. Initialize variables 2. Create Log file stream and configuration file 3. Get List of Coverage Reports and mark test ID’s 4. Parse all Coverage XML reports and store each test case data in

TestCase class objects 5. Each test case object data has a map of Classes Hit, Lines Hit

and Branches Hit 6. BuildTestSuiteTable => Optimize the test case data and

generate a hash map if Overlap Coverage Data 7. Approximate the test case overlap coverage data by grouping

similar test cases which have same amount of line and branch covegare.

8. Generate results using mathematical approximation techniques

Inputs required by the tool Test Cases written in Selenium and exported in

TestNG format

Expected Coverage Constraints

Configuration for the application to be tested –

config.properties

Output obtained Test Case IDs which need to be run to obtain required

line and branch coverage

Implementation - Part I Use of TestNG methods

BeforeClass

AfterClass

AfterMethod

Test

Selenium Test cases exported in TestNG

Cobertura for report generation

Implementation - Part I

Selenium Test Cases in TestNG

format

Execution of test cases one by one

Creation of Cobertura reports

after each test

Replace cobertura.ser file

Map test cases with respective

test IDs

Store the coverage report in xml

format

Implementation - Part II GetTestCaseData – Entry point (Main class)

AppMetaData – Store all the application meta data

TestCase – Store individual test case data

BuildOverlapTable – Store the overlap coverage for each test case with

all other test cases

BuildTestSuiteMap – Store all the indiviual test case data in a Map

BuildClassMap – Store individual class data for each test case

ParseXML – Pares the XML coverage reports and store the data in test

case object

Implementation - Part II

InitializeBuild application

schema

Parse XML coverage reports for each test case

Get all the test case data

Build the overlap coverage map

Select optimum test cases to achieve the

desired coverage

OverlapTable

0 T1 T3 T6 T8

1 T1 T2 T14 T29

2 T3 T7 T12

3 T6 T10

4 T5 T8 T11 T36

5

6

7

Test Case IDs

Gro

up

ID

s

Higest Line/Branch Coverage Test Cases

1 2 3 4

Note: T1, T2, T3 …. T36 are Test Case ID’s

Results generated by manual testing - I

I Test Cases Line Coverage Branch Coverage

INPUT 1, 2, 3, 4, 5, 6 10% 5%

OUTPUT 3, 6 16% -

OUTPUT 1, 2, 3, 4 - 5%

II Test Cases Line Coverage Branch Coverage

INPUT 1 through 61 15% 5%

OUTPUT 3, 6, 9 19% -

OUTPUT 1, 2, 3, 4 - 5%

III Test Cases Line Coverage Branch Coverage

INPUT 1 through 61 22% 10%

OUTPUT 3, 6, 9, 10 19% -

OUTPUT 1, 2, 3, 4, 5, 6 - 6%

IV Test Cases Line Coverage Branch Coverage

INPUT 1 through 61 35% 10%

OUTPUT 3, 6, 9, 10, 11, 14, 16 29% -

OUTPUT 1, 2, 3, 4, 5, 6 - 6%

Results generated by manual testing - II

Results generated by manual testing - II

V Test Cases Line Coverage Branch Coverage

INPUT 1 through 61 40% 11%

OUTPUT 6, 3, 9, 10, 11, 14, 16, 18

36% -

OUTPUT 1, 2, 3, 4, 5, 6 - 6%

VI Test Cases Line Coverage Branch Coverage

INPUT 1 through 61 50% 15%

OUTPUT 3, 6, 9, 10, 11, 14, 16 36% -

OUTPUT 1, 2, 3, 4, 5, 6 - %

Results generated by tool - II Test Cases Line Coverage Branch Coverage

INPUT 1 through 60 10% 5%

OUTPUT 6, 3, 9 11.99% -

OUTPUT 3, 4. 28 - 6.42%

II Test Cases Line Coverage Branch Coverage

INPUT 1 through 60 15% 10%

OUTPUT 3, 6, 9, 10 16.77% -

OUTPUT 3, 4, 5, 6, 28 - 12.37%

Results generated by tool - IIIII Test Cases Line Coverage Branch Coverage

INPUT 1 through 60 22% 15%

OUTPUT 6, 3, 9, 10, 11, 14 23.855% -

OUTPUT 3, 4, 5, 6, 7,8, 28 - 16.71%

IV Test Cases Line Coverage Branch Coverage

INPUT 1 through 60 30% 20%

OUTPUT 6, 3, 9, 10, 11, 14, 16, 18

32.18% -

OUTPUT 3, 4, 5, 6,7,8,9, 28 - 21.44%

Results generated by tool - IIIIII Test Cases Line Coverage Branch Coverage

INPUT 1 through 60 40% 30%

OUTPUT 6, 3, 9, 10, 11, 14, 16, 18, 19 ,20

41.8% -

OUTPUT 3, 4, 5, 6, 7,8, 9, 10, 11, 12, 13, 14, 28

- 30.85%

IV Test Cases Line Coverage Branch Coverage

INPUT 1 through 60 55% 50%

OUTPUT 6, 3, 9, 10, 11, 14, 16, 18, 19 ,20, 21, 23, 24

57.97% -

OUTPUT 28, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,

17, 18, 19, 20, 21

- 50.06%

Observation The tool selects the test cases with higher line and

branch coverage.

The readings are very close to manual testing readings

For most of the scenarios the coverage by manual testing is greater than the tool generated coverage, hence the test cases selected by the tool will always fulfill the expected coverage.

Conclusion The tool selects the test cases with highest line and

branch coverage

The tool makes use of Hash Table and Hash Set to build the overlap coverage to efficiently calculate the expected line and branch coverage.

The tool uses approximation technique which we have implemented to reduce the search space

The tool results show that the total coverage for selected test case will fulfill the expected coverage constraints

Future Scope Test Cases could be directly read from java class files

and then automatically executed

Time constraint can be added for applications which have huge LOC and have more number of test cases

Algorithm can be optimized

Approximation techniques can be tuned to increase precision

The data can be synthesized and stored in a persistent storage like SQL DBMS