code coverage

Code Coveragea stab to the questions of What, When, Why, and How

Aug 2011 the nugget presentationprepared by Ernani Omar Cruz

Thursday, August 25, 11

Usual Developer’s Coding Cycle

Code Test Diagnose Fix

usually a time consuming process

Overall Goal is to minimize time spent on “Fixing”

= COST


What?

is a measure used in software testing. It describes the degree to which the source code of a program has been tested.

Code Coverage

It is a form of testing that inspects the code directly and is therefore a form of white box testing.


Some Types of Code Coverages

• Statement

• Function

• Path

• Condition

• Branch

Statement coverage is a code coverage metric that tells you whether the flow of control reached every executable statement of source code at least once.

Has each function in the program been called? This metric reports whether you invoked each function or procedure.

This metric reports whether each of the possible paths in each function have been followed. A path is a unique sequence of branches from the function entry to the exit.

Condition coverage reports the true or false outcome of each condition.

A branch is the outcome of a decision, so branch coverage simply measures which decision outcomes have been tested.

and more!


Which Do We Use?

It Depends to What We are trying to Code!

But the Point to Note is To Have A Tested Code that it does what is EXPECTED of It to Do.

Accomplish


When?

It Starts with Construction


Early detection of flaws

Higher Confidence

Everyone Happy


Why?


Higher Confidence

Everyone Happy


Why?Why do we need to do this?


Higher Confidence

Everyone Happy



To Ensure Quality


Higher Confidence

Everyone Happy



To Ensure Quality

Minimize Bugs/Defects


Higher Confidence

Everyone Happy



To Ensure Quality

Minimize Bugs/Defects

Avoid Cost due to Rework and Delay


Higher Confidence

Everyone Happy


How much?How much are we going to need to do this properly?

Code coverage of 70-80% is a reasonable goal for system test of most projects with most coverage metrics. Use a higher goal for projects specifically organized for high testability or that have high failure costs.

Minimum code coverage for unit testing can be 10-20% higher than for system testing. --- Bullseye “http://www.bullseye.com/minimum.html”

70-80% Coverage

It Depends.Requires Discretion


http://www.bullseye.com/minimum.html


Why Not 100% Coverage?

Although 100% code coverage may appear like a best possible effort, even 100% code coverage is estimated to only expose about half the faults in a system. Low code coverage indicates inadequate testing, but high code coverage guarantees nothing.

In a large system, achieving 100% code coverage is generally not cost effective. Some reasons are listed below.

• Some test cases are expensive to reproduce but are highly improbable. The cost to benefit ratio does not justify repeating these tests simply to record the code coverage.

• Checks may exist for unexpected error conditions. Layers of code might obscure whether errors in low-level code propagate up to higher level code. An engineer might decide that handling all errors creates a more robust solution than tracing the possible errors.

• Unreachable code in the current version might become reachable in a future version. An engineer might address uncertainty about future development by investing a little more effort to add some capability that is not currently needed.

• Code shared among several projects is only partially utilized by the project under test.Generally, the tester should stop increasing code coverage when the tests become contrived. When you focus more and more on making the coverage numbers better, your motivation shifts away from finding bugs.


“100% code coverage isn't an achievement, it's a minimum requirement. If you write a line of code, you'd better test it.”

- Robert Cecil Martin aka “Uncle Bob”

In the end it all boils down to ROI and pragmatism. Some apps need more testing than others. Some modules need more testing than others. Some bugs need more fixing than others. There will always be a judgment call about whether additional time and money spent on automated testing and coverage are justified or are just a premature optimization.


Roles Each Has to Fill• Project Manager

• Team Lead

• Developer

• Peer

• Tester/QAs

Everyone Has A Responsibility/Stake in Ensuring Quality

The one who allocates resources, shapes priorities, coordinates interactions with project stakeholders and keeps team focused on goals.

The leader of the Development team on the project.

The team member who writes the software and performs the unit tests.

Co-Team member; The one who can help out, review, and check the work of a team member (usually developer)

The one responsible to drive out proper testing and validates functionality


(some) Tools Available that can help us measure/analyze code

coverage• Cobertura

• Clover

• DevPartner

• Emma

• JTest

• Kalistick

• CoView and CoAnt

• Bullseye for C++

• Sonar




• Clover

• DevPartner

• Emma

• JTest

• Kalistick



• Sonar

Cobertura is an open source tool that measures test coverage by instrumenting a code base and watching which lines of code are and are not executed as the test suite runs




• Clover

• DevPartner

• Emma

• JTest

• Kalistick



• Sonar


Some features include historical reporting, complete control over the coverage gathering process, command line toolset and API for legacy integration and more.Clover also allows testing time to be reduced by only running the tests that cover the application code that was modified since the previous build.




• Clover

• DevPartner

• Emma

• JTest

• Kalistick



• Sonar



DevPartner Java Edition (DPJ) integrates a set of functionality enabling developers to analyze Java code for Code Quality and Complexity, Memory Leak Detection, Memory Profiling and Optimization, Performance Profiling and Optimization, Thread Analysis and dead-lock detection, Code Coverage Analysis.




• Clover

• DevPartner

• Emma

• JTest

• Kalistick



• Sonar




EMMA supports class, method, line, and basic block coverage, aggregated at "all", package, source file, class, and method levels. Line and block metrics are offered in "normal" and "weighted" varieties. Other entries in this FAQ provide details on these. EMMA does not do branch or path coverage. However, EMMA reports fractional line coverage to help you visualize untested branches in code.




• Clover

• DevPartner

• Emma

• JTest

• Kalistick



• Sonar





JTest’s basic functionality includes Unit test-case generation, static analysis, regression testing, runtime error detection, and code review.


http://en.wikipedia.org/wiki/Regression_testing




• Clover

• DevPartner

• Emma

• JTest

• Kalistick



• Sonar






A third party application that analyzes the codes with different perspectives that aids users in strategizing release plans.






• Clover

• DevPartner

• Emma

• JTest

• Kalistick



• Sonar







Codign Software is committed to providing a valuable, affordable solution for metrics, automating JUnit tests, mock object creation, path & branch coverage and code testability. We keep our costs low because we only develop features that have benefit and make sense.






• Clover

• DevPartner

• Emma

• JTest

• Kalistick



• Sonar








BullseyeCoverage is a code coverage analyzer for C++ and C that tells you how much of your source code was tested. You can use this information to quickly focus your testing effort and pinpoint areas that need to be reviewed. Code coverage analysis is useful during unit testing, integration testing, and final release.






• Clover

• DevPartner

• Emma

• JTest

• Kalistick



• Sonar









Sonar is an open platform to manage code quality. As such, it covers the 7 axes of code quality: Architecture and Design, Duplications,






• Clover

• DevPartner

• Emma

• JTest

• Kalistick



• Sonar









Sonar is an open platform to manage code quality. As such, it covers the 7 axes of code quality: Architecture and Design, Duplications,comments, coding rules, unit tests, potential bugs, and complexity.




So What do we Use?

Any.


Quality is centralSonar is a web-based application. Rules, alerts, thresholds, exclusions, settings… can be configured online. By leveraging its database, Sonar not only allows to combine metrics altogether but also to mix them with historical measures.

Sonar has got a very efficient way of navigating, a balance between high-level view, dashboard, TimeMachine and defect hunting tools. This enables to quickly uncover projects and / or components that are in Technical Debt to establish action plans.


FeaturesAll projects at a glanceDrill down to source codeCoding rulesUnit TestsStandard MetricsTime MachineMaven ReadyEasinessLeverage existing componentsPluginsSecurity


Software Quality Assessment based on Lifecycle Expectations

The SQALE method is particularly devoted to the management of the Technical debt (or design debt) of AGILE software developments. It allows:

1 To define clearly what creates the technical debt2 To estimate correctly this debt3 To describe this debt into various parts relating to

the testability, the reliability, the changeability, the maintainability… This classification allows to analyze the impact of the debt and to define the priority of code refactoring/remediation activities.

Text


Prepares Us Towards Automated Integration

Having Automated Test Suites


It’s Free.


Thank You.


References• http://en.wikipedia.org/wiki/Code_coverage

• http://www.bullseye.com/statementCoverage.html

• http://www.bullseye.com/paper.html

• http://www.bullseye.com/minimum.html

• http://www.sonarsource.org/

• http://www.atlassian.com/software/clover/

• http://www.bullseye.com/coverage.html

• http://www.codign.com/pathbranchcode.html

• http://software-development-thoughts.blogspot.com/2011/04/how-much-code-coverage-do-you-really.html


http://en.wikipedia.org/wiki/Code_coverage

http://en.wikipedia.org/wiki/Code_coverage

http://www.bullseye.com/statementCoverage.html

http://www.bullseye.com/statementCoverage.html

http://www.bullseye.com/paper.html

http://www.bullseye.com/paper.html



http://www.sonarsource.org/

http://www.sonarsource.org/

http://www.atlassian.com/software/clover/

http://www.atlassian.com/software/clover/

http://www.bullseye.com/coverage.html

http://www.bullseye.com/coverage.html

http://www.codign.com/pathbranchcode.html

http://www.codign.com/pathbranchcode.html

http://software-development-thoughts.blogspot.com/2011/04/how-much-code-coverage-do-you-really.html




References• http://en.wikipedia.org/wiki/Code_coverage#Software_tools

• http://www.codecoveragetools.com/index.php/coverage-process/code-coverage-tools-java.html

• http://emma.sourceforge.net/faq.html#faq-N10042

• http://en.wikipedia.org/wiki/EMMA_(code_coverage_tool)

• http://en.wikipedia.org/wiki/DevPartner

• http://en.wikipedia.org/wiki/Clover_(software)

• http://cobertura.sourceforge.net/

• http://en.wikipedia.org/wiki/Jtest


http://en.wikipedia.org/wiki/Code_coverage#Software_tools

http://en.wikipedia.org/wiki/Code_coverage#Software_tools

http://www.codecoveragetools.com/index.php/coverage-process/code-coverage-tools-java.html




http://emma.sourceforge.net/faq.html#faq-N10042

http://emma.sourceforge.net/faq.html#faq-N10042

http://en.wikipedia.org/wiki/EMMA_(code_coverage_tool)

http://en.wikipedia.org/wiki/EMMA_(code_coverage_tool)

http://en.wikipedia.org/wiki/DevPartner

http://en.wikipedia.org/wiki/DevPartner

http://en.wikipedia.org/wiki/Clover_(software)

http://en.wikipedia.org/wiki/Clover_(software)

http://cobertura.sourceforge.net/

http://cobertura.sourceforge.net/

http://en.wikipedia.org/wiki/Jtest

http://en.wikipedia.org/wiki/Jtest

References

• http://www.kalistick.com/software-radiography-platform.html

• http://www.codign.com/

• http://www.bullseye.com/productInfo.html

• http://www.sqale.org/details


http://www.kalistick.com/software-radiography-platform.html




http://www.codign.com/

http://www.codign.com/

http://www.bullseye.com/productInfo.html

http://www.bullseye.com/productInfo.html

http://www.sqale.org/details

http://www.sqale.org/details

code coverage

Software

code coverage metric

minimum code coverage

code coverage isnt

tested code

lowlevel code

line of code

code directlyand

code coverageis