JAOUT: Automated Generation of

Aspect Oriented Unit Test Guoqing Xu, Z. Yang, H. Huang, Q. Chen, L.

Chen and F. XuSoftware Engineering Lab (SEL)

East China Normal University

APSEC’04, Dec. 2nd 2004, Pusan, Korea

Background

Test Oracle problem -- how to identify oracles in unit test? Automating the test oracle generation -- Manually writing test codes is a labor-intensi

ve job. -- JMLUnit framework [CL02]. Automating the test case generation -- Korat. [BKM02] -- JMLAutoTest [XY03]

Problems

Current test automating process relies on formal assertions and predicts.

Conventional specifications only focus on programs’ functional behaviors, with little support for specifying their non-functional behaviors, e.g. temporal logic, performance...

How to find a common approach to identifying oracles for testing non-functional aspects? How to automate this process?

Our approach

Using a crosscutting property of the program as the criterion to check the correctness of the application in the corresponding aspect is well suited to the unit testing problems.

-- In AOP crosscutting properties are used to model the program from different aspects.

-- Some problems which have been difficult to handle in conventional ways are solved easily.

Application-specific Aspect

A new notion: application-specific aspects (ASS)

-- top-level application related aspects. -- established at the design level. -- may be picked up from low language level

aspects. -- all the ASS for the same application share

some common features, e.g. testing ASS, tracing ASS...

-- may be translated into language level aspects.

Aspect-Oriented Test Description Language

How to describe ASS? -- a formal way is needed. -- can not be too complicated. Aspect-Oriented Test Description Language(AOTDL) -- used by the designer at design level -- can be translated into AspectJ aspects. Basic units in AOTDL -- Utility unit -- MeaninglessCase Advice unit -- Error Advice unit advicetype (arguments): pointcuts: conditions: messag

e

AOTDL (cond.)Class Stack{ public void

init(){...} public void

push ( Node n){...}

...}

TestingAspect tempLogic{

Utility{ protected boolean isInitialized = false; //push is reached pointcut pushReached(Stack st): target(st)&&call(void Stack.push(Integer)); //init is reached pointcut initReached(Stack st): target(st)&&call(void Stack.init(void)); // after advice after(Stack st):initReached (st){ isInitialized = true; }}

AOTDL (cond.)MeaninglessCase Advi

ce{ /*advices for specifyin

g criteria of meaningless test cases */

before(Stack s) : pushReached(s) : s.getSize() >=MAX :

”Overflow”; ... }

Error Advice{ /*advices for specifying cr

iteria of test errors */ before(Stack s): pushReached(s): ! isInitialized : ”Not Ini

tialized” ; ... }}

AOTDL (cond.) //error advices before(Stack s) throws TestErrorException: pushReached(s){ if (!isInitialized){ TestErrorException ex =new TestErrorException(“Not Initialized”); ex.setSource(“TempLogic”); throw ex; }}

JAOUT Framework Overview AOTDL

translate Java

AspectJ

Generate Weave and Compile Test Case Supply

Run the test

An Overview of the basic technique

Testing Aspects

Aspects in AspectJ

Jaout/ Tranlator

Java programs

Bytecode files

Junit test classes

Jaout/ Generator JMLAuto-

Test

Junit test runner

Test Results

Arj

JAOUT: Automated Generation of AO Testing Framework JAOUT takes Java class M.java as the input, and

automatically generate JUnit test framework. -- Aspect_M_Test.java, the JUnit unit test class. -- Aspect_M_TestCase.java, the test case provider. -- Aspect_M_TestClient.java, JMLAutoTest test ca

se generation class.

Test Framework Definition

For class C and its method M(A1 a1, A2 a2…An an), the generated test suite is defined as

--- C[ ] receivers -- A1[ ] vA1; ... ; An[ ] vAn; There is a corresponding init_Ai method for ea

ch type Ai and a method init_receiver in test case provider to initialize test cases.

Testers use APIs provided by JMLAutoTest to generate test case space in test client, and pass it to the test case provider.

Generated Test Method

public void testM(){

for (int i0 = 0; io < receiver.length; i0++){ for (int i1 = 0; i1 < a1.length; i1++){ … try { receiver[i0].M(a1[i1], : : :, an[in]); } catch (MeaninglessCaseException e)

{ /* ... tell framework test case was

meaningless ... */ continue; }

catch (TestErrorException e) { String msg = e.getSource(); fail(msg + NEW LINE + e.getMessa

ge()); } catch (java.lang.Throwable e) { continue; } finally { setUp(); // restore test cases } } }

Test Result ...in push false F Time: 0.06 There was 1 failure: 1) testPush (sample.Stack_Aspect_TestCase)junit.framework.AssertionFailedError: In Test

ing Aspect TempLogic: Not Initilized! at ample.Stack_Aspect_Test.testPush (Stack_Aspect_Test.java:155) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:25) at sample.Stack_Aspect_Test.run(Stack_Aspect_Test.java:26) at sample.Stack_Aspect_TestCase.main (Stack_Aspect_TestCase.java:24) FAILURES!!! Tests run: 3, Failures: 1, Errors: 0, Meaningless:0

Double-phase Testing

A large number of meaningless test inputs in the generated test case space.

It is a waste of time running programs with meaningless inputs.

The effectiveness of test results are compromised if the test is exercised with too many meaningless cases.

Double-phase Testing (Cond.) Goals:1. Prevent meaningless test cases

being processed in the final test, and therefore save the time.

2. Do not require testers to know the details of the program to be tested.

Double-phase Testing (cond.) Steps 1. Establish Operational Profile 2. The first phase test (pre-test) 3. The second phase test (final test) Methods 1. Do the statistics based test cases

selection. 2. Use the pre-test as the cost for reducing

the number of meaningless cases in the final test.

Working Sequence

Operational Profile Operational Profile is the criterion made

by the tester to divide the generated test case space into several partitions.

the validity of double-phase testing relies on the quality of the operational profile.

it is a good idea to start out with several models and evaluate their predictive accuracy before settling on one.

The first phase Take a relatively small number (e.g. 10%) of

test cases out from each partition according to the average distribution.

Run these groups of cases respectively. Make statistics on the number of meaningle

ss test cases appeared in each group.

The second phase Calculate and determine the probability

of meaningless test cases existing in each partition.

Reorganize the test cases according to the inverse proportion of meaningless cases in each partition.

(e.g. take 80% of cases from the partition which produces 20% of meaningless ones in the first phase.)

Run the final test.

Experimental resultsA sample class public class BinaryTree{

public Node root;protected int size;public int getSize() {… }public JMLObjectSequence toObjectSet() {…}

…}

public class Node{ public Node left;

public Node right;public int ID;}

BinaryTree findSubTree(BinaryTree parentTree, Node thisNode)

Testing ASS MeaninglessCase Advice{ before(BinaryTree t) : MethodReached(t) : parentTree==null || thisNode ==null ||

(forall Node n; parentTree.toObjectSet().has(n); n.ID != thisNode.ID)

} }

Test case generation We use JMLAutoTest to generate the test ca

se space of type BinaryTree with a few nodes (five through eight).

We also generate the case space of type Node. It contains 12 nodes whose IDs are from 0 to 11.

Divide the test case space For the test case space of type BinaryTree,

we do not divide it and leave it as the only partition.

For the space of type Node, We divide it into two partitions. The first one contains nodes whose ID varies from 0 to 5 and the second one contains the rest.

A Comparison

nodes in binary

tree

meaningful /total in

final test

time in the first phas

e (s)

total time of the test

(s)

meaningful /total in final test

time in the first phase (s)

total time of the test (s)

5 410/492 0.079

0.266 420/1008 0 0.219

6 1572/1572

0.188

0.422

1584/3168

0 0.468

7 5136/51

36 0.36

0.766 6006/102

96 0 1.25

8 17148/1

7148 0.703

2.016

22880/34

320 0 3.484

Double-phase way Conventional way

Related Work (Spec-based test)

TestEra – Automating OO test generation. [MK01] Mulsaw project, MIT

JMLUnit – Generating test oracles from JML runtime assertions. [CL02] Iowa State Univ.

Korat – Generating test case based on Java predicts. [BKM02] Mulsaw project, MIT.

JMLAutoTest – Generating test framework from JML runtime assertions and test cases based on class invariants. [XY03] SEL, ECNU.

Jov -- java automated unit test based on inferred program properties. [XN03] Univ. of Washington.

Conclusions Traditional formal predicts do not deal with non-funct

ional properties of the program. AOP is well suited to the unit test problems. Designers use AOTDL to build Application-Specific Tes

ting Aspects. JAOUT translates Testing Aspects to AspectJ aspects a

utomatically. JAOUT automatically generates JUnit test framework

and uses the runtime messages thrown by Testing ASS as test oracles.

JAOUT uses double-phase testing approach to filtering out meaningless cases.

Thank you… Questions?