coverage testing with clover - fbkselab.fbk.eu/swat/slide/clover.pdf · coverage testing with...

Coverage testing with clover

Software Analysis and TestingCu Nguyen Duy (cunduy at fbk dot eu)Alessandro Marchetto (marchetto at fbk dot eu)Paolo Tonella (tonella at fbk dot eu)Mariano Ceccato (ceccato at fbk dot eu)

Academic Year 2011-2012

1Thursday, November 17, 11

mailto:[email protected]








Coverage testing (recall)

• Coverage measures describe the degree to which a program has been tested

• tell the effectiveness in terms of coverage of the test set, help improve software quality

• inform quantitatively the project manager about the progress of testing

• Many type of coverage measures

• statements

• paths

• methods, classes

• requirement specifications, etc.


Coverage tool features

• Compile time: automatic instrument code for coverage recording

• Runtime: structure-based coverage recording

• lines, blocks, conditions, methods and classes

• Reporting

• quantitative coverage measure

• percentage of code executed

• most executed v.s. never executed code

• visual navigation

• quickly navigate to code that is not executed to improve the test set


Coverage tools

source code in not-widely-used languages on a variety ofplatforms [30].

3.2. Instrumentation overhead

Coverage-testing tools capture coverage information by moni-toring program execution. Execution is monitored by insertingprobes into the program before or during its execution.A probe is typically a few lines of code that, when executed,generate a record or event that indicates that programexecution has passed through the point where the probe islocated. There are two kinds of overhead associated withinstrumenting a program with probes: the off-line overheadof inserting probes into the program, and the run time over-head of executing the probes to record the execution trace.

3.2.1. Off-line program analysis and instrumentationoverhead

Source code instrumentation, used by most of the toolsincluding BullseyeCoverage, Parasoft Insure!!, Intel CodeCoverage Tool, Semantic Designs and TestWork, requiresrecompilation, but provides more direct results and is moreadaptable to a wide variety of processors and platforms. Itcannot be used when the source code is not available, as isoften the case for third party code. C/C!! tools such asDynamic Memory Systems’ Dynamics, use runtime instru-mentation, which makes them feasible in a productionenvironment. They may be more efficient in terms of compi-lation time, but less portable. The Java coverage tool Koalog

Code Coverage does not require instrumentation, and there-fore no recompilation is needed [25]. It operates with the pro-duction binaries using the Java Debug Interface, which is partof the Java Platform Debugger Architecture (JPDA). KoalogCode Coverage is platform independent, but requires aJPDA compliant Java Virtual Machine (JVM). Agitar’s Agita-tor runs the code in a modified JVM, also using a dynamicinstrumentation approach. eXVantage uses source code instru-mentation for C/C!! and bytecode instrumentation for Java.As compared to the other 16 tools, it has the highest off-lineinstrumentation overhead because it analyzes the program insuch a way that it can select the least number of probes tobe inserted into the target program.

3.2.2. Run-time instrumentation overheadCompanies that provide tools for system software or embeddedsoftware tend to focus more on reducing run-time overhead, sothat their tools can be usable in real-time environments, e.g.CodeTEST [18]. TCAT claims that its TCAT C/C!!Version 3.2 maintains its overhead for execution size ratio at1.1–1.8 and execution speed ratio at 1.1–1.5 [29]; SemanticDesigns claims 1.1–1.3, varying according to language andcompiler, among the best in our survey. Clover claims thattheir execution speed overhead is highly variable, dependingon the nature of the application under test, and the nature ofthe tests. Typical execution speed ratio is 1.2–1.5. eXVantagehas different versions for different platforms, but claims aratio of 1.01 for versions optimized real-time, in some environ-ments, based on initial trials (Table 2) [24].

3.3. Additional features

Coverage testing tools can be used to assist in debugging, andsome of the coverage tools provide debugging assistance, suchas Agitar, Dynamic, JCover, Jtest and Semantic Designs. Eachuses a different solution. For example, Agitar provides a snap-shot and stack trace to help developers to track the cause ofbugs. JCover has the ability to do coverage differencing andcomparison to expose the erroneous code. Semantic Designsprovides slicing and dicing operations on test coverage datavia the GUI to allow code executed/not executed by arbitrarycombinations of test runs to be easily isolated visually.eXVantage uses a dynamic execution slicing approach. Itcreates an execution slice for each test case and reads resultsfrom a testing oracle to generate a bug localization reportautomatically whenever a failed test is detected [8].Coverage testing tools can also be used for program profil-

ing to identify heavily executed parts of programs. Profilingdata can be used in compiler optimization, program refactor-ing, performance-related debugging, etc. Many tools, includ-ing eXVantage, CodeTEST, Dynamic Code Coverage,JCover, PurifyPlus and Semantic Designs, support this feature.

TABLE 1: coverage tools and the languages to which they apply(alphabetical by tool name)

Tool name C!!/C Java Other

Agitar [14] X

Bullseye [15] X

Clover [16] X .net

Cobertura [17] X

CodeTEST [18] X

Dynamic [19] X

EMMA [20] X

eXVantage [21] X X

Gcov [22] X

Intel [23] X FORTRAN

JCover [24] X

Koalog [25] X

Parasoft (C!!test) [26] X

Parasoft (Jtest) [26] X

PurifyPlus [27] X X Basic, .net

Semantic Designs (SD)

[28]

X X C#, PHP,

COBOL,

PARLANSE

TCAT [29] X X

of 9 YANG et al.

THE COMPUTER JOURNAL, 2007

source: doi:10.1093/comjnl/bxm021


Coverage tools



Coverage tools


CodePro AnalytiXhttp://code.google.com/javadevtools/codepro


http://code.google.com/javadevtools/codepro


Coverage tools


CodePro AnalytiXhttp://code.google.com/javadevtools/codepro

Cloverhttp://www.atlassian.com/software/clover




http://www.atlassian.com/software/clover

http://www.atlassian.com/software/clover

Clover

• Is a commercial coverage tool

• Integration with a wide range of development environment: Ant, Eclipse, Maven, IntelliJ

• Installation

• Eclipse update: http://update.atlassian.com/eclipse/clover/

• Get an evaluation license


http://update.atlassian.com/eclipse/clover/




Clover license setting


Enable Clover to an Eclipse

• Right click on the project

• Select Clover > Enable on this project


Clover Eclipse views


Run Unit test with Clover

• In the Coverage Explorer view

• Right click on the project

• Select Run With Clover As > JUnit Test


Clover coverage report



report menu



report menu

projectinfo



report menu

projectinfo

overall coverage



report menu

projectinfo

overall coverage

testresults



report menu

projectinfo

overall coverage

testresults

coverage by package



report menu

projectinfo

overall coverage

testresults

tree mapcoverage by package


Exercise 1

• Open project: Exercise-4-CurrentAccount

• Enable Clover on this project

• Execute Unit Test

• Generate coverage report

• Report the test coverage of each class, identify which lines of code have not executed

• Task: improve the test cases to reach 100% coverage


Exercise 2

• Similar to exercise 1, with Exercise-3-MathOperators


coverage testing with clover - fbkselab.fbk.eu/swat/slide/clover.pdf · coverage testing with...

Documents