OCAT: Object Capture based Automated Testing

Hojun Jaygarl, Carl K. ChangIowa State University

Sunghun KimThe Hong Kong University of Science and Technology

Tao XieNorth Carolina State University

ISSTA 2010

Problem

2

Generating object inputs is hard

in Object-Oriented (OO) unit testing

Automated Test Generation (ATG)

3

Automatically generate test inputs for a unit.

Reduce manual efforts in OO unit testing.

Two Main Types of ATG Techniques

Direct object construction Directly assign values to object fields of the object

instance E.g., Korat [ISSTA'02]

Method sequence generation Generate method sequences that can produce an

object instance under construction E.g., Randoop [ICSE’07], Pex [TAP'08]

Results of the State of the Arts in ATG

5

Pex Pex automatically generates test inputs.

Dynamic Symbolic Execution

21% of branch coverage [Thummalapenta et al. FSE '09] for QuickGraph, a C# graph library.

Randoop Randoop automatically generates method sequences

Random but feedback directed

58% of branch coverage [Thummalapenta et al. FSE '09]

45% according to our evaluation for Apache Common Collections 3.2

Cause of not-covered branches

# of branches

Explanation

Insufficient object 135 (46.3%) unable to generate desirable object instances required to cover certain branches.

String comparison 61 (20.9%) Difficult to randomly find a desirable string to satisfy such constraints, since the input space of string type values is huge.

Container object access

39 (13.4%) Not easy to create a certain size of a container with necessary elements.

Array comparison 25 (8.6%) Not easy to create a certain size of an array with necessary elements.

Exception branches 18 (6.1%) These branches have a particularity of exception handling code that handles run-time errors

Environmental setting

9 (3.1%) Hard to get environment variables and file-system structure.

Non-deterministic branch

4 (1.3%) Hard to handle multi-threading and user interactions

Case of not-covered branches

6

Almost 90%

Run Randoop for three projects Apache Commons, XML Security, and JSAPWe randomly selected 10 source code files from each subject and investigated the causes of uncovered branches.

Not-Covered Branch Example

7

Checks the

validity of “doc”

Korat Require manual efforts for writing class invariants

and value domains.

Pex Pex uses built-in simple heuristics for generating

fixed sequences, which are often ineffective.

Randoop Random approach cannot generate relevant

sequences that produce desirable object instances.

Limitations of Current Approaches

8

Practical approach (high coverage)

Reflect real usage

Easier process

Idea

9

Object capture based Automated Testing

OCAT Overview

10

Capturing Process

Instrument the target program

Capture objects From program execution From system test execution

11

Instrumentation

12

Byte code instrumentation

Serialization

13

Get a type and a concrete state of captured

objects.

Prune redundant objects.

A concrete state representation for the state-

equality checking [Xie et al. ASE’04]

Serialized Object Example

14

A list that has bank accounts

Object Generation

15

Feed captured objects to an existing automated

method sequence generation technique.

Object Capturin

g

Object Capturin

g

Method Sequenc

esGenerati

on

Method Sequenc

esGenerati

on

Execution

ExecutionCaptured

objects

Sequences

Generated objects

Usage of Captured Objects

16

Captured objects are de-serialized and used as

test inputs.

Evolved Objects

Indirect usage

Captured instance of

A

foo(A) returns A

Captured instance of

A

bar(A)returns B

New instance

of B

New instance of

A

Method Sequence with Captured Objects

17

Evaluation - Setup

18

Evaluation - Captured Objects

19

Q1 How much can OCAT improve code coverage through captured object instances?

19.0% improved from 45.2%

28.5% and 17.3% improved from 29.6% and 54.1%

Evaluation- Captured Objects

20

Object Mutation

Approach – Object Mutation (MTT)

22

AbstractReferenceMap

Evaluation- Mutated Objects

23

Q2 How much can mutated object instances further improve code coverage?

Evaluation- Total

24

25.5% improved on average, with maximum 32.7%22.9

%

32.7%

20.9%

Why is It a Feasible Idea?

25

Reflect real usage

Potential for being desirable inputs in achieving new branch coverage

Capturing objects is easy

Discussion Object Capturing Process

Problem: OCAT’s coverage depends on captured objects

Capturing objects is an easy process (but we still need to capture “good-enough” objects).

Captured Objects and Software Evolution Problem: software evolves and objects are changing. Captured object instances may be obsolete and not

be valid anymore.

26

Discussion

Branches to Cover Problem: Still not-covered branches are more

than 20%. Cross-system object capturing

objects can be captured from system A and used for system B.

Static analysis currently we use a simple static analysis.

Iterative process two phases, object generation and object mutation,

can be iteratively applied

27

Threats to Validity

Software under test might not be representative Our three subjects may yield better or worse OCAT

coverage than that of other software projects.

Our object capturing relies on the existing tests OCAT test coverage reported in this paper

depends on the quality and quantity of existing tests.

28

Conclusions Problem

Hard to generate desirable object instances in OO unit testing.

OCAT approach Capture objects from program execution. Generate more objects with method sequence

generation. Mutate directly the captured objects to try to

cover those not-yet-covered branches.

29

Conclusions Results

OCAT helps Randoop to achieve high branch coverage averagely 68.5%

25.5% improved (with maximum 32.7%) from only 43.0% achieved by Randoop alone.

Future work Enhance static analysis part Apply captured objects to other state-of-the-art

techniques (e.g., parameterized unit testing, concolic testing, and model checking approaches).

Release the tool.30

31

Thank you!