Combinatorial Interaction Testing for TestSelection in Grammar-Based Testing

Elke Salecker, Sabine Glesner

Technical University of Berlin, Germany

Workshop on Combinatorial Testing (CT) 2012Montreal, Canada

Introduction Background Approach Evaluation Conclusions

Motivation

Grammar-based Testing

• black-box testing approach• test data derived from context-free grammar

à automatic generation of test data

Problem

• exhaustive testing of test sets infeasible• controlled/random-based enumaration process• rule combination coverage not in focus

New Approach

à test selection based on combinatorial interaction testing

à automatic generation of CIT test specification

Elke Salecker, Sabine Glesner 1

Introduction Background Approach Evaluation Conclusions

Motivation

Grammar-based Testing

• black-box testing approach• test data derived from context-free grammar

à automatic generation of test data

Problem

• exhaustive testing of test sets infeasible• controlled/random-based enumaration process• rule combination coverage not in focus

New Approach

à test selection based on combinatorial interaction testing

à automatic generation of CIT test specification

Elke Salecker, Sabine Glesner 1

Introduction Background Approach Evaluation Conclusions

Motivation

Grammar-based Testing

• black-box testing approach• test data derived from context-free grammar

à automatic generation of test data

Problem

• exhaustive testing of test sets infeasible• controlled/random-based enumaration process• rule combination coverage not in focus

New Approach

à test selection based on combinatorial interaction testing

à automatic generation of CIT test specification

Elke Salecker, Sabine Glesner 1

Introduction Background Approach Evaluation Conclusions

Outline

1 Introduction

2 CIT and Grammars

3 Test Set Generation

4 Evaluation

5 Conclusions

Elke Salecker, Sabine Glesner 2

Introduction Background Approach Evaluation Conclusions

Combinatorial Interaction Testing

CIT specification:parameters, values

P1 P2 P3 P4a1 b1 c1 d1a2 b2 c2 d2a3

test case (a1, b1, c1, d2)

test set P1 × P2 × P3 × P4, 24 test casescoverage criterion (t=2)

constraints ¬(b1 ∧ c2)

Elke Salecker, Sabine Glesner 3

Introduction Background Approach Evaluation Conclusions

Combinatorial Interaction Testing

CIT specification:parameters, values

P1 P2 P3 P4a1 b1 c1 d1a2 b2 c2 d2a3

test case (a1, b1, c1, d2)

test set P1 × P2 × P3 × P4, 24 test casescoverage criterion (t=2)

constraints ¬(b1 ∧ c2)

Elke Salecker, Sabine Glesner 3

Introduction Background Approach Evaluation Conclusions

Combinatorial Interaction Testing

CIT specification:parameters, values

P1 P2 P3 P4a1 b1 c1 d1a2 b2 c2 d2a3

test case (a1, b1, c1, d2)

test set P1 × P2 × P3 × P4, 24 test cases

coverage criterion (t=2)

constraints ¬(b1 ∧ c2)

Elke Salecker, Sabine Glesner 3

Introduction Background Approach Evaluation Conclusions

Combinatorial Interaction Testing

CIT specification:parameters, values

P1 P2 P3 P4a1 b1 c1 d1a2 b2 c2 d2a3

test case (a1, b1, c1, d2)

test set P1 × P2 × P3 × P4, 24 test casescoverage criterion (t=2)

1 a3 b1 c1 d22 a2 b2 c2 d23 a1 b2 c1 d24 a1 b1 c2 d15 a2 b1 c1 d16 a3 b2 c2 d1

constraints ¬(b1 ∧ c2)

Elke Salecker, Sabine Glesner 3

Introduction Background Approach Evaluation Conclusions

Combinatorial Interaction Testing

CIT specification:parameters, values

P1 P2 P3 P4a1 b1 c1 d1a2 b2 c2 d2a3

test case (a1, b1, c1, d2)

test set P1 × P2 × P3 × P4, 24 test casescoverage criterion (t=2)

1 a3 b1 c1 d22 a2 b2 c2 d23 a1 b2 c1 d24 a1 b1 c2 d15 a2 b1 c1 d16 a3 b2 c2 d1

constraints ¬(b1 ∧ c2)

Elke Salecker, Sabine Glesner 3

Introduction Background Approach Evaluation Conclusions

Grammars

nonterminals N = {�, r, imm}terminals Σ = {Const, ObjAddr, Content, Plus, Assign}

rules def: �→ Assign((ObjAddr a), r)(lhs → rhs) add: r → Plus(r, r)

addimm: r → Plus(r, imm)use: r → Content(ObjAddr a)r2c: r → Const cr2i: r → immi2c: imm → Const c

start symbol �derivation sequence of rule applications

< add, addimm, r2c >

r Plus

r r

Plus

Plus

r imm

r

Plus

Plus

Const 2 imm

r

Elke Salecker, Sabine Glesner 4

Introduction Background Approach Evaluation Conclusions

Grammars

nonterminals N = {�, r, imm}terminals Σ = {Const, ObjAddr, Content, Plus, Assign}rules def: �→ Assign((ObjAddr a), r)(lhs → rhs) add: r → Plus(r, r)

addimm: r → Plus(r, imm)use: r → Content(ObjAddr a)r2c: r → Const cr2i: r → immi2c: imm → Const c

start symbol �

derivation sequence of rule applications< add, addimm, r2c >

r Plus

r r

Plus

Plus

r imm

r

Plus

Plus

Const 2 imm

r

Elke Salecker, Sabine Glesner 4

Introduction Background Approach Evaluation Conclusions

Grammars

nonterminals N = {�, r, imm}terminals Σ = {Const, ObjAddr, Content, Plus, Assign}rules def: �→ Assign((ObjAddr a), r)(lhs → rhs) add: r → Plus(r, r)

addimm: r → Plus(r, imm)use: r → Content(ObjAddr a)r2c: r → Const cr2i: r → immi2c: imm → Const c

start symbol �derivation sequence of rule applications

< add, addimm, r2c >

r

Plus

r r

Plus

Plus

r imm

r

Plus

Plus

Const 2 imm

r

Elke Salecker, Sabine Glesner 4

Introduction Background Approach Evaluation Conclusions

Grammars

nonterminals N = {�, r, imm}terminals Σ = {Const, ObjAddr, Content, Plus, Assign}rules def: �→ Assign((ObjAddr a), r)(lhs → rhs) add: r → Plus(r, r)

addimm: r → Plus(r, imm)use: r → Content(ObjAddr a)r2c: r → Const cr2i: r → immi2c: imm → Const c

start symbol �derivation sequence of rule applications

< add, addimm, r2c >

r Plus

r r

Plus

Plus

r imm

r

Plus

Plus

Const 2 imm

r

Elke Salecker, Sabine Glesner 4

Introduction Background Approach Evaluation Conclusions

Grammars

nonterminals N = {�, r, imm}terminals Σ = {Const, ObjAddr, Content, Plus, Assign}rules def: �→ Assign((ObjAddr a), r)(lhs → rhs) add: r → Plus(r, r)

addimm: r → Plus(r, imm)use: r → Content(ObjAddr a)r2c: r → Const cr2i: r → immi2c: imm → Const c

start symbol �derivation sequence of rule applications

< add, addimm, r2c >

r Plus

r r

Plus

Plus

r imm

r

Plus

Plus

Const 2 imm

r

Elke Salecker, Sabine Glesner 4

Introduction Background Approach Evaluation Conclusions

Grammars

nonterminals N = {�, r, imm}terminals Σ = {Const, ObjAddr, Content, Plus, Assign}rules def: �→ Assign((ObjAddr a), r)(lhs → rhs) add: r → Plus(r, r)

addimm: r → Plus(r, imm)use: r → Content(ObjAddr a)r2c: r → Const cr2i: r → immi2c: imm → Const c

start symbol �derivation sequence of rule applications

< add, addimm, r2c >

r Plus

r r

Plus

Plus

r imm

r

Plus

Plus

Const 2 imm

r

Elke Salecker, Sabine Glesner 4

Introduction Background Approach Evaluation Conclusions

Outline

1 Introduction

2 CIT and Grammars

3 Test Set Generation

4 Evaluation

5 Conclusions

Elke Salecker, Sabine Glesner 5

Introduction Background Approach Evaluation Conclusions

Symmetry in Derivations

r2c r → Const ci2c imm → Const cadd r → Plus(r, r)addimm r → Plus(r, imm)

derivation:

add ⇒ addimm ⇒ r2c ⇒ i2c ⇒ r2c

Plus

Plus

Const 2 Const 2

Const 4

derivation:

add ⇒ r2c ⇒ addimm ⇒ r2c ⇒ i2c

Plus

Const 4 Plus

Const 2 Const 2

Elke Salecker, Sabine Glesner 6

Introduction Background Approach Evaluation Conclusions

Symmetry in Derivations

r2c r → Const ci2c imm → Const cadd r → Plus(r, r)addimm r → Plus(r, imm)

derivation:

add ⇒ addimm ⇒ r2c ⇒ i2c ⇒ r2c

Plus

Plus

Const 2 Const 2

Const 4

derivation:

add ⇒ r2c ⇒ addimm ⇒ r2c ⇒ i2c

Plus

Const 4 Plus

Const 2 Const 2

Elke Salecker, Sabine Glesner 6

Introduction Background Approach Evaluation Conclusions

Principle of Specification Generation

Derivations with Fixed Length1 2 3 4 5 6

1 def add add r2c r2c r2c2 def add use add r2c r2c3 def add r2c add r2c r2c. . .6 def add addimm r2c i2c use7 def add addimm use i2c use. . .30 def add use add use use

P1 P2 P3 P4 P5 P6derivation length = number of parametersvalue of parameter i = rule can be applied in step iconstraints for invalid combinations

Elke Salecker, Sabine Glesner 7

Introduction Background Approach Evaluation Conclusions

Principle of Specification Generation

Derivations with Fixed Length1 2 3 4 5 6

1 def add add r2c r2c r2c2 def add use add r2c r2c3 def add r2c add r2c r2c. . .6 def add addimm r2c i2c use7 def add addimm use i2c use. . .30 def add use add use use

P1 P2 P3 P4 P5 P6derivation length = number of parametersvalue of parameter i = rule can be applied in step iconstraints for invalid combinations

Elke Salecker, Sabine Glesner 7

Introduction Background Approach Evaluation Conclusions

CIT Specification Generation

1 Generate compact representation of derivations

2 Calculate value sets

3 Calculate constraints

Elke Salecker, Sabine Glesner 8

Introduction Background Approach Evaluation Conclusions

Generate Compact Representation of Derivations

Recursive Construction:

Base Case derivation length = 1store for terminal ruleslhs nonterminal, rule identifier

Recursion derivation length > 1

check remaining rulessplit overall length for number of nonterminals in rulefind smaller derivationsstore lhs nonterminal, rule identifier, extended pattern

Elke Salecker, Sabine Glesner 9

Introduction Background Approach Evaluation Conclusions

Generate Compact Representation of Derivations

Recursive Construction:

Base Case derivation length = 1store for terminal ruleslhs nonterminal, rule identifier

Recursion derivation length > 1check remaining rulessplit overall length for number of nonterminals in rulefind smaller derivationsstore lhs nonterminal, rule identifier, extended pattern

Elke Salecker, Sabine Glesner 9

Introduction Background Approach Evaluation Conclusions

Generate Compact Representation of Derivations

def:�→Assign((ObjAddr a),r)add:r → Plus(r, r)addimm:r → Plus(r, imm)use:r → Content(ObjAddr a)r2c:r → Const cr2i:r → immi2c:imm → Const c

L NT Rule RHS Pattern

1 r use ∅r2c ∅

imm i2c ∅

2 r r2i < (1, imm) >

� def < (1, reg) >

3 r addimm < (1, reg), (1, imm) >

add < (1, r), (1, r) >

� def < (2,r) >

4 add < (1, r), (2, r) >

< (2, r), (1, r) >

Elke Salecker, Sabine Glesner 10

Introduction Background Approach Evaluation Conclusions

Generate Compact Representation of Derivations

def:�→Assign((ObjAddr a),r)add:r → Plus(r, r)addimm:r → Plus(r, imm)use:r → Content(ObjAddr a)r2c:r → Const cr2i:r → immi2c:imm → Const c

L NT Rule RHS Pattern

1 r use ∅r2c ∅

imm i2c ∅2 r r2i < (1, imm) >

� def < (1, reg) >

3 r addimm < (1, reg), (1, imm) >

add < (1, r), (1, r) >

� def < (2,r) >

4 add < (1, r), (2, r) >

< (2, r), (1, r) >

Elke Salecker, Sabine Glesner 10

Introduction Background Approach Evaluation Conclusions

Generate Compact Representation of Derivations

def:�→Assign((ObjAddr a),r)add:r → Plus(r, r)addimm:r → Plus(r, imm)use:r → Content(ObjAddr a)r2c:r → Const cr2i:r → immi2c:imm → Const c

L NT Rule RHS Pattern

1 r use ∅r2c ∅

imm i2c ∅2 r r2i < (1, imm) >

� def < (1, reg) >

3 r addimm < (1, reg), (1, imm) >

add < (1, r), (1, r) >

� def < (2,r) >

4 add < (1, r), (2, r) >

< (2, r), (1, r) >

Elke Salecker, Sabine Glesner 10

Introduction Background Approach Evaluation Conclusions

Generate Value Sets

Recursive Construction:

Base Case derivation length = 1collect rules for length 1

Recursion derivation length > 1iterate over rules with considered nonterminalextend parameter with rulesfor each rule iterate over alternativesdescend for each nonterminal

Elke Salecker, Sabine Glesner 11

Introduction Background Approach Evaluation Conclusions

Generate Compact Representation of Derivations

L NT Rule RHS Pattern

1 r use ∅r2c ∅

imm i2c ∅2 r r2i < (1, imm) >

� def < (1, r) >

3 r addimm < (1, r), (1, imm) >

add < (1, r), (1, r) >

� def < (2,reg) >

4 r addimm < (2, r), (1, imm) >

add < (1, r), (2, r) >

< (2, r), (1, r) >

� def < (3,r) >

P1P2P3P4

Elke Salecker, Sabine Glesner 12

Introduction Background Approach Evaluation Conclusions

Generate Compact Representation of Derivations

L NT Rule RHS Pattern

1 r use ∅r2c ∅

imm i2c ∅2 r r2i < (1, imm) >

� def < (1, r) >

3 r addimm < (1, r), (1, imm) >

add < (1, r), (1, r) >

� def < (2,reg) >

4 r addimm < (2, r), (1, imm) >

add < (1, r), (2, r) >

< (2, r), (1, r) >

� def < (3,r) >

P1 defP2P3P4

Elke Salecker, Sabine Glesner 12

Introduction Background Approach Evaluation Conclusions

Generate Compact Representation of Derivations

L NT Rule RHS Pattern

1 r use ∅r2c ∅

imm i2c ∅2 r r2i < (1, imm) >

� def < (1, r) >

3 r addimm < (1, r), (1, imm) >

add < (1, r), (1, r) >

� def < (2,reg) >

4 r addimm < (2, r), (1, imm) >

add < (1, r), (2, r) >

< (2, r), (1, r) >

� def < (3,r) >

P1 defP2 addimmP3P4

Elke Salecker, Sabine Glesner 12

Introduction Background Approach Evaluation Conclusions

Generate Compact Representation of Derivations

L NT Rule RHS Pattern

1 r use ∅r2c ∅

imm i2c ∅2 r r2i < (1, imm) >

� def < (1, r) >

3 r addimm < (1, r), (1, imm) >

add < (1, r), (1, r) >

� def < (2,reg) >

4 r addimm < (2, r), (1, imm) >

add < (1, r), (2, r) >

< (2, r), (1, r) >

� def < (3,r) >

P1 defP2 addP3 use, r2cP4

Elke Salecker, Sabine Glesner 12

Introduction Background Approach Evaluation Conclusions

Generate Compact Representation of Derivations

L NT Rule RHS Pattern

1 r use ∅r2c ∅

imm i2c ∅2 r r2i < (1, imm) >

� def < (1, r) >

3 r addimm < (1, r), (1, imm) >

add < (1, r), (1, r) >

� def < (2,reg) >

4 r addimm < (2, r), (1, imm) >

add < (1, r), (2, r) >

< (2, r), (1, r) >

� def < (3,r) >

P1 defP2 addimmP3 use, r2cP4 i2c

Elke Salecker, Sabine Glesner 12

Introduction Background Approach Evaluation Conclusions

Generate Constraints

Recursive Construction:

Base Case derivation length = 1disjunction of all rules for length 1

Recursion derivation length > 1iterate over rules with considered nonterminal

construct disjunctionfor each rule iterate over alternatives

construct disjunctionfor each alternative

construct conjunction

Elke Salecker, Sabine Glesner 13

Introduction Background Approach Evaluation Conclusions

Generate Compact Representation of Derivations

L NT Rule RHS Pattern

1 r use ∅r2c ∅

imm i2c ∅2 r r2i < (1, imm) >

� def < (1, r) >

3 r addimm < (1, r), (1, imm) >

add < (1, r), (1, r) >

� def < (2,reg) >

4 r addimm < (2, r), (1, imm) >

add < (1, r), (2, r) >

< (2, r), (1, r) >

� def < (3,r) >

Elke Salecker, Sabine Glesner 14

Introduction Background Approach Evaluation Conclusions

Generate Compact Representation of Derivations

L NT Rule RHS Pattern

1 r use ∅r2c ∅

imm i2c ∅2 r r2i < (1, imm) >

� def < (1, r) >

3 r addimm < (1, r), (1, imm) >

add < (1, r), (1, r) >

� def < (2,reg) >

4 r addimm < (2, r), (1, imm) >

add < (1, r), (2, r) >

< (2, r), (1, r) >

� def < (3,r) >

def −→ f

Elke Salecker, Sabine Glesner 14

Introduction Background Approach Evaluation Conclusions

Generate Compact Representation of Derivations

L NT Rule RHS Pattern

1 r use ∅r2c ∅

imm i2c ∅2 r r2i < (1, imm) >

� def < (1, r) >

3 r addimm < (1, r), (1, imm) >

add < (1, r), (1, r) >

� def < (2,reg) >

4 r addimm < (2, r), (1, imm) >

add < (1, r), (2, r) >

< (2, r), (1, r) >

� def < (3,r) >

def −→(addimm ∧ f )

Elke Salecker, Sabine Glesner 14

Introduction Background Approach Evaluation Conclusions

Generate Compact Representation of Derivations

L NT Rule RHS Pattern

1 r use ∅r2c ∅

imm i2c ∅2 r r2i < (1, imm) >

� def < (1, r) >

3 r addimm < (1, r), (1, imm) >

add < (1, r), (1, r) >

� def < (2,reg) >

4 r addimm < (2, r), (1, imm) >

add < (1, r), (2, r) >

< (2, r), (1, r) >

� def < (3,r) >

def −→(addimm ∧((use ∨ r2c) ∧ f))

Elke Salecker, Sabine Glesner 14

Introduction Background Approach Evaluation Conclusions

Generate Compact Representation of Derivations

L NT Rule RHS Pattern

1 r use ∅r2c ∅

imm i2c ∅2 r r2i < (1, imm) >

� def < (1, r) >

3 r addimm < (1, r), (1, imm) >

add < (1, r), (1, r) >

� def < (2,reg) >

4 r addimm < (2, r), (1, imm) >

add < (1, r), (2, r) >

< (2, r), (1, r) >

� def < (3,r) >

def −→(addimm ∧((use ∨ r2c) ∧ i2c))

Elke Salecker, Sabine Glesner 14

Introduction Background Approach Evaluation Conclusions

Outline

1 Introduction

2 CIT and Grammars

3 Test Set Generation

4 Evaluation

5 Conclusions

Elke Salecker, Sabine Glesner 15

Introduction Background Approach Evaluation Conclusions

Size of Parameters

P1 P2 P3 P4 P5 P6

G1 Spec3 5 17 9

Spec4 10 38 26 13

Spec5 10 43 38 26 13

Spec6 12 55 47 38 26 13

G2 Spec3 5 21 13

Spec4 10 44 30 20

Spec5 10 53 47 32 20

Spec6 12 88 67 54 32 20

G186 rules12 initial rulesG2110 rules12 initial rules

à generation within seconds

Elke Salecker, Sabine Glesner 16

Introduction Background Approach Evaluation Conclusions

Size of Test Sets

G1 G2 G3 G4

L 5 (1) 6 (2) 6 (1) 7 (2)

6 Spec 1 23 42 24 42 1 22 3 52 23 3 52

Derivations 64 128 144 288

TCases(ACTS) 16 25 13 22

8 Spec 1 23 44 24 44 1 22 3 54 23 3 54

Derivations 640 1280 2160 4320

TCases(ACTS) 22 32 17 28

10 Spec 1 23 46 24 46 1 22 3 56 23 3 56

Derivations 7168 14336 36288 -

TCases(ACTS) 32 38 24 30

• significant reduction• generation failed for case study grammars

Elke Salecker, Sabine Glesner 17

Introduction Background Approach Evaluation Conclusions

Size of Test Sets

G1 G2 G3 G4

L 5 (1) 6 (2) 6 (1) 7 (2)

6 Spec 1 23 42 24 42 1 22 3 52 23 3 52

Derivations 64 128 144 288

TCases(ACTS) 16 25 13 22

8 Spec 1 23 44 24 44 1 22 3 54 23 3 54

Derivations 640 1280 2160 4320

TCases(ACTS) 22 32 17 28

10 Spec 1 23 46 24 46 1 22 3 56 23 3 56

Derivations 7168 14336 36288 -

TCases(ACTS) 32 38 24 30

• significant reduction• generation failed for case study grammars

Elke Salecker, Sabine Glesner 17

Introduction Background Approach Evaluation Conclusions

Conclusion and Future Work

Conclusion

• new approach for grammar-based testing• criterion guarantees t-wise rule coverage• test selection based on combinatorial interaction testing• CIT specification automatically generated

• very encouraging results regarding test set size

Future Work

• enhancement of constraint generation• combination with alternative approach for grammar-based

testing

• comprehensive case study for compiler back ends

Elke Salecker, Sabine Glesner 18

Introduction Background Approach Evaluation Conclusions

Conclusion and Future Work

Conclusion

• new approach for grammar-based testing• criterion guarantees t-wise rule coverage• test selection based on combinatorial interaction testing• CIT specification automatically generated

• very encouraging results regarding test set sizeFuture Work

• enhancement of constraint generation• combination with alternative approach for grammar-based

testing

• comprehensive case study for compiler back ends

Elke Salecker, Sabine Glesner 18

IntroductionCIT and GrammarsTest Set GenerationEvaluationConclusions