EdSketch: Execution-Driven Sketching for Java

Jinru Hua and Sarfraz Khurshid

University of Texas at Austin

7/14/2017

Execution-Driven Sketching - Motivation

● A sketch is a partial program with holes, which can be completed by automated tools.

● Existing program sketching approaches translate sketch to SAT, which may lead to impractical problem to translate all libraries or create models for libraries.

● We present execution-driven sketching, a synthesis approach using backtracking search.

● Our key novelty is to introduce effective pruning strategies to explore actual behaviors with libraries.

2

Outline

● Overview and Motivation

● Motivating Example

● Approach

● Evaluation

● Summary

3

class LinkedList { Entry head; public void reverse() { Entry ln1= ; Entry ln2=null; Entry ln3=null; while ( ) { } } }

reverse(): reverse the singly linked list

Motivating Example - Program Sketch

Original Input Expected Output

next1 2 3

head

3 2 1

head

4

@Testpublic void testThreeEntries() { LinkedList list = new LinkedList(new int[]{1,2,3}); list.reverse(); assertEquals(“[3,2,1]”, list.toString());}

@Testpublic void testThreeEntries() { LinkedList list = new LinkedList(new int[]{1,2,3}); list.reverse(); assertEquals(“[3,2,1]”, list.toString());}

class LinkedList { Entry head; public void reverse() { Entry ln1=EdSketch.EXP(...); Entry ln2=null; Entry ln3=null; while (EdSketch.COND(...)) { EdSketch.ASSIGN_BLOCK(...); } } }

reverse(): reverse the singly linked list

Motivating Example - A user-written sketch

Original Input Expected Output

next1 2 3

head

3 2 1

head

5

class LinkedList { Entry head; public void reverse() { Entry ln1=EdSketch.EXP(...); Entry ln2=null; Entry ln3=null; while (EdSketch.COND(...)) { EdSketch.ASSIGN_BLOCK(...); } } }

reverse(): reverse the singly linked list

Motivating Example - A complete program

Original Input Expected Output

next1 2 3

head

3 2 1

head

6

1. EdSketch.EXP(...):head

2. EdSketch.COND(...):

ln1 != null

3. EdSketch.ASSIGN_BLOCK(...):

head = ln1; ln1 = head.next;head.next = ln2;ln2 = head;

Outline

● Overview and Motivation

● Motivating Example

● Approach

● Evaluation

● Summary

7

Architecture - Expr. Gen. & Instrumentation

8

Approach - Expression Generation

● Generate field dereferences for all visible variables within a predefined bound of field deref., e.g., 1.

● Assign a unique identifier for each candidate.- E.g., head = ln1 is represented as [0, 1].- A 4-stmt block is represented as [0,1,1,4,4,2,2,0]

head = ln1; ln1 = head.next;head.next = ln2;ln2 = head;

● Non-deterministically select an identifier- int left_id = Verify.getInt(0,7);

9

head ln1 ln2 ln3 head.next ln1.next ln2.next ln3.next

0 1 2 3 4 5 6 7

Instrumentation Example - Assignments

10

public void reverse() {... while (EdSketch.COND(...)) { EdSketch.ASSIGN_BLOCK(...);

}

11

public void reverse() {... while (EdSketch.COND(...)) {+ for (Statement s: EdSketch.ASSIGN_BLOCK(...)) {+ EdSketch_Assign stmt = (EdSketch_Assign) s;+ Entry rhs_val = (Entry) stmt.getRHS();+ switch (stmt.getLHS_id()) {+ case 0: head = rhs_val; break;+ case 1: ln1 = rhs_val; break;

...+ case 7: ln3.next = rhs_val; break;+ }+ } }

0 head

1 ln1

2 ln2

3 ln3

4 head.next

5 ln1.next

6 ln2.next

7 ln3.next

Instrumentation Example - Local var. update

Architecture - Pruning and Backtracking

12

● If a candidate satisfies any rules, EdSketch prunes it without execution against tests.

- “E1 = E1;” 一 omitted (no-op)

- “V1 = V2; V2 = V1;” 一 subsumed by “V1 = V2;”

- “V1 = V2; V1 = V3;” 一 subsumed by “V1 = V3;”

- “V3 = V1; V2 = V1;” and id(V3)>id(V2)- isomorphic to “V2 = V1; V3 = V1;”

Approach - Assignment pruning rules

13

Condition Pruning Strategies

● Condition Candidate Generation

○ Operators that combine expressions, e.g., ln1 != null■ Primitive types:■ Non-primitive types:

○ Based on symmetry, we combine an expression with the ones with bigger identifiers. ■ only consider e==f and e!=f, ignore f==e and f!=e.

○ Two constant values: true, false (e==e and e!=e)

● Condition Value Grouping○ Split condition candidates based on evaluated value (true or

false)14

● Stateful backtracking using Java PathFinder (JPF)● Stateless backtracking inspired by VeriSoft

○ Using Explorer class from Juzi [Suen-UT-MS-05]

Approach - Backtracking Engines

15

Outline

● Overview and Motivation

● Motivating Example

● Approach

● Evaluation

● Summary

16

Evaluation - Subjects

● 10 data structure subjects, bounded exhaustive test suites, avg. 9 test cases (wrote oracles manually).

17

Type Name Test 1 2 3 4

A BSTAS 8 196 38.4 K 7.5 M 1.5 B

A MEDAS 7 16 256 4.1 K 66 K

A LLREV 4 64 4.1 K 262.4 K 16.7 M

A RBTAS 15 676 457 K 309 M 209 B

A DLLAF 4 196 38.4 K 7.5 M 1.5 B

A DLLAL 4 196 38 K 7.5 M 1.5 B

C BSTCD 8 392 190.5 K 74.7 M 29.3 B

C RBTRM 15 74 5.5 K 7.4 M 10.0 B

C MEDCD 7 96 9.2 K 884.9 M 84.9 M

C RBTCD 15 74 5.5 K 7.4 M 10.0 B

● Sketch system is a SAT-based synthesizer

● Manually translate subjects to Sketch language

- Translate the sketch, e.g., while ({|(head|ln1|...|ln3.next) (==|!=) (head|ln1|...|ln3.next|null)|})

- Translate the test suite, e.g., harness void test() { LinkedList l = new LinkedList(...); ... //list “1,2,3” reverse(l.head); assert l.head.val == 3; ... }

Evaluation - Comparison with Sketch

18

Evaluation - Performance

● Comparison with Sketch synthesizer

19

Evaluation - Performance

● Comparison with Sketch synthesizer

20

EdSketch-JVM can sometimes even be faster than the SAT-based Sketch synthesizer.

Evaluation - EdSketch Pruning Strategies

● Comparison of number of executed programs with and without pruning strategies.

21

Evaluation - EdSketch Pruning Strategies

● Comparison of number of executed programs with and without pruning strategies.

22

Assignment Pruning--Avg. reduction rate: 21%Condition Pruning--Avg. reduction rate: 56%

Evaluation - Larger Application● Our approach can handle sketching tasks from Java

applications. - A task from Apache commons-lang with 55 KLOC code and

3,274 test cases.

23

public String random(int end, char[] chars,...) {

if (chars != null) {

end = chars.length;

}...

}

/* Program sketch after manual transformation */

public String random(int end, char[] chars,...) {

if (EdSketch.COND(...)) {

EdSketch.ASSIGN_BLOCK(...);

}...

}

Evaluation - Advanced Features● Our approach can sketch programs with reflection,

I/O, and native calls.

24

public boolean repOK() throws Exception {...

LinkedList<Node> workList=new LinkedList<Node>();

workList.add(root);

while (!workList.isEmpty()) {

Node cur= workList.removeFirst();

Node exp=(Node) EdSketch.EXP(...);

//expect current.left

if(Node.class.getField("left").get(exp)!=null) {

...//omit the rest

}

}

}

Related Work

25

● Some related program synthesis approaches:○ Sketching partial programs [Sketch’06, JSketch’15]

○ Synthesizing API chains [Prospector’05, CodeHint’14, SyPet’17]

○ Embedding angelic choice construct with backtracking solver but no pruning strategies. [Angelic Programming’10]

● Some related strategies in program repair approaches:○ Assignment pruning rules are similar to adaptive equivalence

for program repair [AE’13]

○ Condition value grouping is similar to abstract condition values for repair [SPR’15]

Summary

● We introduce a novel execution-driven approach to sketch Java programs using backtracking search.

● We introduce effective pruning strategies to explore actual behaviors with libraries.

● The experimental results show that the performance of EdSketch compares well with SAT-based Sketch synthesizer based on our subjects.

● EdSketch can complete some sketches in presence of complex constructs.

26

27

Backup Slides

28

Partial Expression Syntax

● EdSketch.COND(...):- EdSketch.COND(Entry.class, new String[]{“head”,

“ln1”, “ln2”, “ln3”}, new Object[]{head, ln1, ln2, ln3});

● EdSketch.BLOCK(...):- EdSketch.BLOCK(Entry.class, new String[]{“head”,

“ln1”, “ln2”, “ln3”}, new Object[]{head, ln1, ln2, ln3});

● Set the number of assignment statements:- EdSketch.BLOCK(...).setLength(4);

29

Approach - Condition Value Grouping

● Split condition candidates based on evaluated value (true or false)

30

Instrumentation - Handle infinite loop

31

+ private int count = 0; public void reverse() {... while (EdSketch.COND(Entry.class)) {+ for (Statement s: EdSketch.ASSIGN_BLOCK(Entry.class)) {+ EdSketch_Assign stmt = (EdSketch_Assign) s;+ Entry rhs_val = (Entry) stmt.getRHS();+ switch (stmt.getLHS_id()) {+ case 0: head = rhs_val; break;+ case 1: _ln1_ = rhs_val; break;

...+ case 7: _ln3_.next = rhs_val; break;+ }+ }+ if (count++ > EdSketch.LOOPBOUND)+ EdSketch.backtrack(); }

0 head

1 ln1

2 ln2

3 ln3

4 head.next

5 ln1.next

6 ln2.next

7 ln3.next

EdSketch-JPF vs. EdSketch-JVM

32

EdSketch-JPF EdSketch-JVM

● Quick to implement, off-the-shelf enabling technology

● Efficient

● Real backtracking:○ Handles JVM memory-

level side effects

● Resets relevant parts of state○ Assumes running the

same JUnit suite again gives the same outcome

● Can be extended to sketch multithreaded code

● Handles native code, complex libraries

Discussion

● The efficacy of our pruning strategy is consistent in finding the first solution and all solutions.

● The order of test cases has negligible influence on sketching performance based on our subjects.

● We only compare the performance with one state-of-the-art synthesizers on our dataset.

● We manually translate Sketch program and tests.● Our bounded exhaustive test suite may still lead to

plausible sketching results, i.e., passing all tests but are incorrect based on manual inspection.

33

Sketch Procedure

34

EdSketch Condition Candidate Generation

35

EdSketch Value Grouping Algorithm

36

● If a candidate satisfies any rules, EdSketch prunes it without execution against tests.

- E1 = E2

- V1 = V2, V2 = V1

- V1 = V2, V1 = V3

- V3 = V1, V2 = V1 & id(V3)>id(V2) - isomorphic to V2 = V1, V3 = V1

Approach - Assignment pruning rules

37

ln1.next = ln1.next

ln1 = ln2, ln2 = ln1

ln1 = ln2, ln1 = ln3

ln3 = ln1, ln2 = ln1

Evaluation - File I/O● Our approach can sketch programs with reflection,

I/O, and native calls.

38

public int getSum(String file) {

Scanner scan = new Scanner(new File(file));

int a = scan.nextInt();

int b = scan.nextInt();

//expect a+b

return (int) EdSketch.EXPOP(new double[]{a,b});

Evaluation - Native Calls● Our approach can sketch programs with reflection,

I/O, and native calls.

39

public native String nativeToString(int x, int y);

public String swap(int x, int y) {

int tmp = x;

//expect x = y, y = tmp

EdSketch.ASSIGN_BLOCK(int.class);

String str = nativeToString(x,y);

return str;

}

Counter-example-driven synthesis

40

Synthesize control c that completes S for a random input; check if the control c also works for all other inputs. If not, add counter example input and repeat

Reference:A.Solar-Lezama, L.Tancau, R.Bodik, V.Saraswat, S.Seshia. Combinatorial Sketching for Finite Programs. ASPLOS’06, page 404-415.

How automated is EdSketch?

To create: conditions (“exp rop exp”) and assignment statements (“exp = exp;”), where rop ∈ {<, >, ≤, ≥, ==, !=}At the algorithm level:

Inputs: type T for exp, bound B for number of dereferences, bound S on number of statements

At the implementation level:Inputs: T, B, S, root pointers (Map<VarName, Var>)Manual instrumentation:

Inserting new fields to model local variablesInserting non-deterministic choices (e.g., switch-case statements) 41

Reference - Program Synthesis[Sketch’06] A.Solar-Lezama, L.Tancau, R.Bodik, S.Seshia, V.Saraswat. Combinatorial sketching for finite programs. In ASPLOS, pages 404–415, 2006.

[JSketch’15] J.Jeon, X.Qiu, J.Foster, A.Solar-Lezama. JSketch: sketching for Java. In FSE, pages 934–937, 2015.

[Prospector’08] D.Mandelin, L.Xu, R.Bodik, D.Kimelman. Jungloid mining: helping to navigate the API jungle. In PLDI, pages 48–61, 2005.

[CodeHint’14] J.Galenson, P.Reames, R.Bodik, B.Hartmann, K. Sen. CodeHint: dynamic and interactive synthesis of code snippets. In ICSE, pages 653–663, 2014.

[SyPet’17] Y.Feng, R.Martins, Y.Wang, I.Dillig, T.Reps. Component-based synthesis for complex apis. In POPL, pages 599–612, 2017.

[Perelman et al.’12] D.Perelman, S.Gulwani, T.Ball, D.Grossman. Type-directed completion of partial expressions. In PLDI, pages 275–286, 2012.

42

Reference - Program Synthesis (contd.)[InSynth’10] V.Kuncak, M.Mayer, R.Piskac, P.Suter. Complete functional synthesis. In PLDI, pages 316–329, 2010.

[Slang’14] V.Raychev, M.Vechev, E.Yahav. Code completion with statistical language models. In PLDI, pages 419– 428, 2014.

[Bodik et al.’10] R.Bodik, S.Barman, S.Chandra, J.Galenson, D.Kimelman, C.Rodarmor, and N.Tung. Programming with angelic nondeterminism. In POPL, pages 339–352, 2010.

43