java pathfinder jpf tutorial - //javapathfinder.sourceforge.net/ test input generation with java...

Java Pathfinder

JPF Tutorial - http://javapathfinder.sourceforge.net/

Test Input Generation With Java Pathfinder – Visser, Pasareanu, Khurshid

Generalized Symbolic Execution for Model Checking and Testing – Khurshid, Pasareanu, Visser

Symbolic Execution and Model Checking for Testing - Pasareanu, Visser

JPF–SE - A symbolic execution extension to Java PathFinder – Anand, Pasareanu, Visser

JPF is an explicit state software model checker for Java bytecode

Usage

• As an exhaustive testing tool

• As a model checker

• Deploys abstractions and optimizations

• Extensibility– Search-/VMListeners

– Model Java Interface (MJI)

– Configurable ChoiceGenerators

– Bytecode Factories (New, no tutorials yet)

What’s New• Symbolic Test Data Generation The symbolic execution extension uses a BytecodeFactory to

override JPF's core bytecodes to generate concrete test cases. In a nutshell, this works by using the JPF field/stackframe attribute system to collect symbolic path conditions, which are then fed into a contraint solver to obtain concrete test data. The choice points in this execution mode are branch instructions.

• Symbolic Threadsafety Test Generation This is a very simple "symbolic" execution mode that tries to identify potential thread safety problems, and then creates code to test this with concrete value JPF execution mode. The symbolic information is much more simple than with the symbc extension, and can be kept in the execution trace. The first implemented test category detects un-synchronized access of fields from within the same public (exported) method, to identify candidates for the PreciseRaceDetector listener.

• Compositional Verification Framework The CV extension implements a machine learning algorithm that can be used for assume/guarantee reasoning, to partition a system into components that can be verified separately. Ultimately, the goal of this extension is to dramatically improve the scalability of JPF, but it can be also used to generate environment assumptions for the UML model checking, to determine "valid" event sequences. This extension is mostly implemented with JPF listeners.

• Numeric Property Verification This bytecode replacement extension started as a set of numeric instruction classes to detect over/underflow, but got extended to also include inexact value propagation (NaN, Inf), exact floating point comparison, and potential catastrophic cancellation (loss of precision by subtraction/addition)

• UML State Chart Model Checking The state chart framework is a variation of the user interface model checking, for the purpose of model checking UML state chart diagrams. It is a combination of MJI libraries that are based on a unique translation scheme for UML state charts, together with generic applications (test drivers) to execute these state charts. The framework supports both testing and model checking, using the same scripting language for environment specification.

Symbolic Execution and

Test-input GenerationWillem Visser

&Corina Pasareanu and Peter Mehlitz

RIACS/Kestrel Technology/CSCNASA Ames Research Center

[email protected]

6

Overview

• Motivation for Model Checking Programs• Introduction to Java PathFinder• Symbolic Execution• Coverage based Test-input generation• Conclusions

7

Motivation

Ariane 501

Mars Polar Lander

Software Errors can be very costly

8

More Recently

Software problem with Spirit

9

Model Checking

OK

Error trace

orFinite-state model

Temporal logic formula

Model Checker

Line 5: …Line 12: …Line 15:…Line 21:…Line 25:…Line 27:… …Line 41:…Line 47:…

10

Overview


11

Key Points

• Models can be infinite state– Unbounded objects, threads,…– Depth-first state generation (explicit-state)– Verification requires abstraction

• Handle full Java language – mostly only for closed systems– Cannot handle native code

• no input/output through GUIs, files, Networks, …• must be modeled by java code instead

• Allows Nondeterministic Environments– JPF traps special nondeterministic methods

• Checks for user-defined assertions, deadlock and LTL properties• Incorporates a number of search strategies

– DFS, BFS, A*, Best-first, etc.• http://ase.arc.nasa.gov/jpf

12

Overview


13

Concrete Execution Path (example)

x = 1, y = 0

1 >? 0

x = 1 + 0 = 1

y = 1 – 0 = 1

x = 1 – 1 = 0

0 – 1 >? 0

int x, y;

if (x > y) {

x = x + y;

y = x – y;

x = x – y;

if (x – y > 0)

assert(false);

}

14

Symbolic Execution Tree (example)

x = X, y = Y

X >?

Y

[ X > Y ] y = X + Y – Y = X

[ X > Y ] x = X + Y – X = Y

[ X > Y ] Y - X >? 0

[ X <= Y ] END [ X > Y ] x = X + Y

[ X > Y, Y – X <= 0 ] END [ X > Y, Y – X > 0 ] END

int x, y;

if (x > y) {

x = x + y;

y = x – y;

x = x – y;

if (x – y > 0)

assert(false);

}

15

Forward Symbolic Execution

• technique for executing a program on symbolic input values

• explore program paths– for each path, build a path condition

– check satisfiability of path condition

• state– symbolic values of variables, path condition, and counter

• various applications– test generation

– program verification

• traditional use– programs with fixed number of int variables

16

Challenges in Generalizing Symbolic Execution

• how to handle fields in dynamic structures?

• how to handle aliasing?

• how to generate tests?– satisfy criteria– satisfy precondition– are in-equivalent

17

Example

class Node {int elem;Node next;

Node swapNode() { if (next != null) if (elem > next.elem) { Node t = next; next = t.next; t.next = this; return t; } return this;}

}

? null

E0 E1

E0

E0 E1 null

E0 E1 ?

E0 E1

E0 E1

Input list + Constraint Output list

E0 > E1

none

E0 <= E1

none

E0 > E1

E0 > E1

E0 > E1

E1 E0 ?

E1 E0

E1 E0

E1 E0 null

E0 E1

E0

? null

NullPointerException

18

Generalized Symbolic Execution

• model checker generates and explores “symbolic” execution tree– path conditions are added at each branch point

• off-the-shelf decision procedures check path conditions• model checker backtracks if not satisfiable

– non-determinism handles aliasing• explore different heap configurations explicitly

– concurrency

• lazy initialization– initializes program’s inputs on an “as-needed” basis– no a priori bound on input sizes

• preconditions to initialize inputs only with valid values

19

Algorithm (aliasing)

• when method execution accesses field fif (f is uninitialized) { if (f is reference field of type T) { non-deterministically initialize f to null a new object of class T (with uninitialized fields) an object created during prior field initialization (alias) } if (f is numeric/string field) initialize f to a new symbolic value }

20

Algorithm (illustration)

E0next

E1next

tnull

tE0

nextE1

next?

nextE0

nextE1

t next E0 nextE1

next

t

E0next

E1next

t

consider executingnext = t.next;

Precondition: acyclic list

E0 E1next

tnull

next

tE0 E1

next?

nextnext

21

program instrumentation

counterexample(s)/test suite[heap+constraint+thread scheduling]

Implementation via Instrumentation

model checking

decision procedure

instrumented program

correctness specification

continue/backtrack

path condition (data)heap configurationthread scheduling

state:

original

program

22

Overview

• Motivation for Model Checking Programs• Introduction to Java PathFinder• Symbolic Execution• Coverage based Test-input generation (TIG)• Conclusions

23

White- & Black-Box Testing

void add(Object o) { buffer[head] = o; head = (head+1)%size;}

Object take() { … tail=(tail+1)%size; return buffer[tail];}



RequirementsSpecification

Input Generator

Input Generator

Oracle

Oracle

Testing CriteriaCoverage of

Specification & Code

Testing CriteriaSpecification Coverage

24

Functional Spec

White- & Black-Box Testing





Input Generator

Input Generator

Oracle

Oracle


Specification & Code

Testing CriteriaSpecification Coverage

Input Spec

Acyclic Linked List

After removing the last element the list is empty

Adding to a full list is not allowed

25

Model Checking & TIG

OK

Error trace

orExecutable Specification

Property specifying coverage cannot be achieved

Model Checker

Line 5: …Line 12: …Line 15:…Line 21:…Line 25:…Line 27:… …Line 41:…Line 47:…

No test-inputcan achieve

desired coverage

Test-input to achieve coverage

26

Test-Input Generation (TIG)with Symbolic Execution

• … is it still true?• White-box versus black-box

– Symbolic execution most often white-box

• Simple data is straightforward• Complex data

– Black-box is (reasonably) straightforward – Korat (ISSTA’02)– White-box?

“… symbolic execution for testing programs is a more exploitable technique in the short term than the more general one of program verification”

James KingCACM 19:7, 1976

27

Functional Spec

White- & Black-Box Testingfor Complex Data





Input Generator

Input Generator

Oracle

Oracle


Input Specification & Code

Testing CriteriaInput Specification

Coverage

Input Spec

Class Invariant

Pre-condition to every method

boolean repOk()

no runtime errors exist

28

Red-Black Trees

(1) The root is BLACK

(2) Red nodes can only have black children

(3) All paths from a node to its leaves contain the same number of black nodes.

Self-balancing Binary Search TreesJava SortedMap Implementation

(4) Acyclic

(5) Consistent Parents

repOk(): conditions (1)-(5)

29

repOk() Fragment

boolean repOk(Entry e) { // root has no parent, root is black,… // RedHasOnlyBlackChildren workList = new LinkedList(); workList.add(e); while (!workList.isEmpty()) { Entry current=(Entry)workList.removeFirst(); Entry cl = current.left; Entry cr = current.right; if (current.color == RED) { if(cl != null && cl.color == RED) return false; if(cr != null && cr.color == RED) return false; } if (cl != null) workList.add(cl); if (cr != null) workList.add(cr); } // equal number of black nodes on left and right sub-tree… return true;}

30

Black-box TIG

• Generate inputs based on analysis of input structure– e.g. Rover plan generation, Korat

• 100% “coverage” of input structures up to a predetermined upper-bound– e.g. all red-black trees with 6 or less nodes

• Complex data requires that only valid structures be considered– Use class invariant to reduce number of input structures to consider

• a predicate characterizing all the instances of a class• boolean repOk()

• Check code coverage using generated structures as input

• Advantage – test code for which no source is available

31

Symbolic execution for black-box TIG

• Symbolic execution of repOk()– Generate new structures only when repOk() returns true– Limit the size of the structures generated– Only correct structures will be generated

• repOk() returns true after all nodes in the tree have been visited, hence they must all be concrete

• symbolic (partial) structures can fail repOk()

• Similar to Korat– Except we can also deal with data constraints

32

Symbolic Execution of repOk()Example

public static boolean repOk(Entry e) {

if (e == null) return true;

if (e.color == RED) return false;

…

33

White-box TIG

• Consider code coverage criterion when generating test inputs

• Challenge– Treating complex data with symbolic execution

• Use repOk() as a method precondition during symbolic execution of source code:– use repOk() to convert “symbolic” input structures into

concrete structures that cover the code and pass repOk()– use repOk() also to eliminate “symbolic” structures during

lazy initialization, thus reducing the search space

34

repOk() x 2abstract and concrete

Symbolic Execution of Code During Lazy Initializationcheck Abstract repOK()

When coverage is achieved, solve the symbolic constraints

to create concrete inputs

To concretize inputs by symbolic execution of

Concrete repOk()over symbolic structures

- as with Black-box TIG -

35

White-box TIG: cover branches in deleteEntry(Entry p)

/* precondition: p.repOk() */

private void deleteEntry(Entry p) {

if (p.left != null && p.right != null) {

Entry s = successor(p);

swapPosition(s, p);

}

Entry replacement = (p.left != null ? p.left : p.right);

if (replacement != null) {

replacement.parent = p.parent;

if (p.parent == null)

root = replacement;

else if (p == p.parent.left) {

p.parent.left = replacement;

}

else

p.parent.right = replacement;

p.left = p.right = p.parent = null;

if (p.color == BLACK)

fixAfterDeletion(replacement); ...

36

Symbolic Execution for white-box TIG

if (p.left != null && p.right != null) { ...

Symbolic structurebefore executingbranch

Concretize

Concrete structurethat will cover the code

The symbolic structure is used as input to repOk() and lazily executed to obtain the concrete structure

Symbolic structure(s)that coverthe branch

This structure “passes”the abstract repOk()

37

Conservative repOk()

• Used to eliminate symbolic structures that cannot be converted to a concrete structure that satisfy repOk() and therefore do not describe valid inputs

• Because of symbolic structures we use abstraction– conservative_RepOk() can return TRUE, FALSE or Don’t Know

• if FALSE, ignore that structure by backtracking• if TRUE or Don’t Know, continue ...

• Example: (2) Red nodes have only black children.

FALSE TRUE Don’t Know

38

Conservative repOk()

// root has no parent, root is black,… // RedHasOnlyBlackChildren workList = new LinkedList(); workList.add(e); while (!workList.isEmpty()) { Entry current=(Entry)workList.removeFirst(); Entry cl = current.left; Entry cr = current.right; if (current.color == RED) {

if(current._left_is_initialized && cl != null && cl.color == RED) return false;

if(current._right_is_initialized && cr != null && cr.color == RED) return false; }

if (current._left_is_initialized && cl != null) workList.add(cl);

if (current._right_is_initialized && cr != null) workList.add(cr); } // equal number of black nodes on left and right sub-tree… return true;

39

Cover branches in deleteEntry(Entry p)

/* precondition: p.repOk() */

private void deleteEntry(Entry p) {

if (p.left != null && p.right != null) {

Entry s = successor(p);

swapPosition(s, p);

}

Entry replacement = (p.left != null ? p.left : p.right);

if (replacement != null) {

replacement.parent = p.parent;

if (p.parent == null)

root = replacement;

else if (p == p.parent.left) {

p.parent.left = replacement;

}

else

p.parent.right = replacement;

p.left = p.right = p.parent = null;

if (p.color == BLACK)

fixAfterDeletion(replacement);...

40

Lazy Initialization from Partial Structures

Partial structure satisfying conservative_RepOk()

Solution that satisfies repOk() Not a solution!

ConcretizationBy lazy initialization of

repOK()

41

Black Box Results

196

84

36

16

8

2

Tests

8810086794233(16)2926

8810086183017(6)605

8890864329(4)154

3

2

1

N

50

5

0

fixD%BC

88721035(2)5.5

668243(2)3.2

61851(1)3

fixIns%BC

delEnt%BC

CandidateStructures

StructsTime

Size 7: Korat 256753 candidates vs 35804

42

White-box

11(53)

Tests

100

fixD%BC

8886110627.392

fixIns%BC

delEnt%BC

CandidateStructures

MemTime

43

Conclusions

• Other JPF features– Partial-order reductions– Observations

• Test-input Generation– Examples with primitive data as well as complex data– Make link with Shape Analysis

• Derive conservative repOk() from concrete repOk() automatically

• Symbolic Execution– Invariant generation

• Combining Test-input generation and runtime monitoring– X9 testing framework for a next generation Mars Rover

java pathfinder jpf tutorial - //javapathfinder.sourceforge.net/ test input generation with java...

Documents