AUTOMATING THE PROCESS FOR BUILDING RELIABLE SOFTWARE

Simon Watson, Field Applications Engineer

Who is this presentation for?

Companies that: Have a safety or mission critical application Have requirements to do unit and integration

testing Have requirement to prove code coverage Realise that manual testing is error prone and

expensive Realise that NOT testing is not an option Need the ability to automate regression testing

© 2009 Vector Software Inc.

Automating the process for building reliable software

“Industry wide, 50% of a software budget is used for FAA, level A, software structural testing.”

Mark Hall, Lockheed Martin, Test Manager

Agenda

Who are we

Unit & Integration Test Automation

Traceability

Automated regression testing

Software Development Philosophies

Who is Vector Software

Vector Software is an international company, established in 1990 in RI, USA

We develop VectorCAST, a tool suite to automate unit, integration, and system testing for native and embedded environments

We deliver verifiable savings of time and money

© 2009 Vector Software Inc.

Manual Unit & Integration Testing

Determine whichunits to stub

Build stubs Build test driverfor the unit (main)

DebugCompile test harness(Test driver + stubs)

Modify harness components for next test case

Determine level ofsource code coverage

Build test reportsfor individual test cases

Gather test data resultsinto readable format

Executing test harness

Time to prepare for testing

Driver Code needs to be created

Stubs Need to be written

Maintenance of Stubs and Drivers: For example, Windows NT 4 had:

6 million lines of application code 12 million lines of test code

Test cases are often as likely or more likely to contain errors than the code being tested

Steve McConnell, Code Complete

Testing,etc.25%

Preparingto test75%

Common testing pitfalls

Pitfalls Each developer decides

how to test their components No common practices so

difficult to peer review

Work is done manually or with home-grown tools Manually = time-consuming Home-grown tools have

limited capabilities

Results Effort is difficult to

coordinate Difficult to know how

expensive it is

Scripts cannot be reused from phase to phase Unit test scripts

abandoned when integrating

Regression testing difficult

Automating the process

Automatically build Test framework for the Unit(s) Under Test Parser - determine data / call dependencies between units Build Test Driver - invoke subprograms within UUT Stubbing – automatic stubbing based on specifications or

real code

Construct Test Cases Intuitive Interface - construction utility with type/data

understanding Basis Path Analysis - identify execution paths for 100%

code coverage

Record & Report Test Data Execution Management - Run Test Cases generated Test Reporting - Standards Compliant Coverage Analysis – Structural, Branch, MC/DC

Unit and Integration Testing

Unit(s) Under Test

TestDriver

Units(.c, .cpp, Ada pkg)

Source Code

StubbedDependents

Real Dependents(.obj files)

VectorCAST Builds Test Harnesses Automatically© 2009 Vector Software Inc.

Embedded Unit/Integration Testing

Code Analysis, Structural Code Coverage

When to stop testing? If some code is not tested, you need more tests?

Testing without measuring code coverage exercises 55% of code (Robert O’Grady, HP)

Why measuring code coverage is important: Identify unexecuted program statements Verify that each path is taken at least once

Coverage appears to be “useful”

BUT: what is Structural Code Coverage?Coverage Sample

Entry

13

25

4

Exit

Types of Coverage

Statement coverage The executable lines of source code are being

encountered as the program runs Statement != line of code Careful about the following:

int* p = NULL;if (condition) p = &variable;*p = 123;

100% coverage with 1 test will miss this pointer bug

Types of Coverage (cont.)

Branch coverage The outcomes have occurred for each branch point in

the program Reports whether Boolean expressions tested in control

structures evaluated to both true and false. Careful about the following code:

if (condition1 && (condition2 || function1())) statement1;else statement2;

Could potentially consider this control structure completely exercised without the call to function1()

Types of Coverage (cont.)

Modified Condition/Decision Coverage Similar to Branch coverage and statement coverage Also reports on the values of all Boolean components of

a complex conditional For example: if (a && b) is a complex conditional with two

terms MC/DC requires enough test cases to verify every

condition that can affect the result of its encompassing decision

Created at Boeing and is required for aviation software for DO-178B certification.

What does 100% coverage prove?

There are classes of errors which certain coverage types cannot determine:

int * h = 0;

if (x){

h = &(a[1]);

}

if (y){

return *h;

}

return 0;

100% coverage with (x, y) set to (1, 1) and (0, 0) But, (0, 1) => OOPS!? Dereference fail Therefore does, 100% coverage imply 100% tested? No!

Tests for 100% Coverage

Tests for 100% Bugs

Basis Path Analysis

Decision outcomes within a software function should be tested independently Errors based on interactions between decision outcomes are more

likely to be detected

Expansion of cyclomatic complexity metric basis path tests cases required = cyclomatic complexity

Complexity is correlated with errors testing effort is concentrated on error-prone software.

Minimum required number of tests is known in advance testing process can be planned and monitored in greater detail than

with other testing strategies

Basis paths must be linearly independent

Basis Path Analysis, an example

Basis Paths

Traceability

Why is traceability important 100% Code coverage ≠ 100% Application Complete 100% Test Cases Passed ≠ 100% Application Complete 100% Requirements passed ≠ 100% Application Complete

100% Code coverage + 100% Test Cases Passed +100% Requirements passed

= 100% Application Complete

There needs to be an efficient way to measure and link in real time: System and Derived Requirements Code Coverage Test Case Results

Traceability: Requirements Gateway

Permit bi-directional flow of data between Requirements Management Tool and Test Case Management Tool

Linking Requirements Intuitive and simple while constructing test cases

For Each Test Case, the linked requirements should show: Requirement Identifier Requirement Description

For Each Requirement, the linked test cases should show: Test Name Test Status {pass | fail | none} Test Coverage {% coverage, type of coverage}

Traceability Example - Test Tool & Requirements Tool Integration

Test Case

System Requirements

Linked Requirements

Automated Regression testing

Why is it important? Cost Reduction

Streamline the functional testing process

Prevent Defect Propagation New defects caught as soon as they are introduced

Project Health Total number of project tests, Percentages of classes and methods, and Pass/failure rates

Automating the regression testing process

Manages the Regression Test Process Execution of test cases Collation of data Across multiple configurations and baselines

Key metrics: number of failures, overall code coverage, total number of tests, and percentage of classes and methods with direct tests

Database Stores data from all regression test runs in an SQL database

Reporting Provides metrics reports at any level from all Unit(s) Under Test. Graphing of test data from prior runs

Regression Testing, an example

Units in regression

test

Regression Testing Reports, an example

Graphic Report View:

Shows a graph based on the results

filtered by Report Options

Software Development Philosophies

There are many approaches to developing systems Agile, Behavior Driven, Design-driven, Lean

software, Rapid application, etc The later we identify a defect in our

system, the costlier it is to fix this defect common solution to this is to introduce

code inspections, or code static analysis

Errors by classification

The scope of most errors is fairly limited 85% of errors can be corrected without modifying more

than one routine (Endres 1975) Most construction errors are the programmers’

fault 95% are caused by programmers, 2% by systems

software (i.e. Compiler and OS), 2% by some other software, and 1% by the hardware (Code Complete)

Therefore, we can draw the conclusion, the sooner we test the programmers software, the sooner we can find the majority of the errors in the system

The cost of fixing a defect

1

10

100

Design BeforeCoding

Coding BeforeTest

Test Field

Co

st F

act

or

Test Driven Development

Derived from test-first programming concepts of extreme programming

Develop test cases prior to developing the application code you are about to write.

Relies on the repetition of a very short development cycle

Can also be used for legacy code projects

The 3 Laws of Test Driven Design

1 You may not write production code until you have written a failing unit test

2 You may not write more of a unit test than is sufficient to fail, and not compiling is failing

3 You may not write more production code than is sufficient to pass the currently failing test

Test Driven Design Flow Chart

Benefits of Test Driven Design

Catch design flaws earlier Design tested at granular level prior to

writing code Easily identify missing requirements Easily identify missing test cases Reduce testing time by a factor of 10 Easy to migrate tests into continuous

integration Cheaper to fix a bug

Requirements for Automating Test Driven Design

Unit test automation tool can be used Unit test automation tool should be

able to handle changes to underlying code base as it matures

Direct link between test case and resulting code coverage

The ability to stub functions in the same module

Stub by Function inside Unit Under Test

In Summary

Typically 50% of the cost of a reliability process is in validation

Validation costs can be reduced by using an automated unit testing tool

Code coverage mixed with basis path testing can improve the quality of software

Traceability between test cases, requirements and code coverage helps ensure a complete application is shipped

Automated regression testing leads to continuous validation

Test Driven Development is a cost effective way of developing reliable systems

Contact Info

Vector Software:8 Grafton St, London, Westminster, W1S 4EL,

United KingdomTel: +44 203 178 6149Email: info@vectorcast.comWeb: www.vectorcast.com

© 2009 Vector Software Inc.