improving test suites via operational abstraction

45
Michael Ernst, page 1 Improving Test Suites via Operational Abstraction Michael Ernst MIT Lab for Computer Science http://pag.lcs.mit.edu/~mernst/ Joint work with Michael Harder, Jeff Mellen, and Benjamin Morse

Upload: sezja

Post on 12-Feb-2016

20 views

Category:

Documents


0 download

DESCRIPTION

Improving Test Suites via Operational Abstraction. Michael Ernst MIT Lab for Computer Science http://pag.lcs.mit.edu/~mernst/ Joint work with Michael Harder, Jeff Mellen, and Benjamin Morse. Creating test suites. Goal: small test suites that detect faults well - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Improving Test Suites via Operational Abstraction

Michael Ernst, page 1

Improving Test Suites viaOperational Abstraction

Michael ErnstMIT Lab for Computer Sciencehttp://pag.lcs.mit.edu/~mernst/

Joint work withMichael Harder, Jeff Mellen, and Benjamin Morse

Page 2: Improving Test Suites via Operational Abstraction

Michael Ernst, page 2

Creating test suitesGoal: small test suites that detect faults wellLarger test suites are usually more effective

• Evaluation must account for sizeFault detection cannot be predicted

• Use proxies, such as code coverage

Page 3: Improving Test Suites via Operational Abstraction

Michael Ernst, page 3

Test case selectionExample: creating a regression test suiteAssumes a source of test cases

• Created by a human• Generated at random or from a grammar• Generated from a specification• Extracted from observed usage

Page 4: Improving Test Suites via Operational Abstraction

Michael Ernst, page 4

ContributionsOperational difference technique for selecting

test cases, based on observed behavior• Outperforms (and complements) other

techniques (see paper for details)

• No oracle, static analysis, or specification

Stacking and area techniques for comparing test suites• Corrects for size, permitting fair comparison

Page 5: Improving Test Suites via Operational Abstraction

Michael Ernst, page 5

OutlineOperational difference technique for selecting

test casesGenerating operational abstractionsStacking and area techniques for comparing

test suitesEvaluation of operational difference techniqueConclusion

Page 6: Improving Test Suites via Operational Abstraction

Michael Ernst, page 6

Operational difference technique

Idea: Add a test case c to a test suite S if c exercises behavior that S does not

Code coverage does this in the textual domainWe extend this to the semantic domainNeed to compare run-time program behaviors

• Operational abstraction: program properties• x > y• a[] is sorted

Page 7: Improving Test Suites via Operational Abstraction

Michael Ernst, page 7

Test suite generationor augmentation

Idea: Compare operational abstractions induced by different test suites

Given: a source of test cases; an initial test suite

Loop:• Add a candidate test case• If operational abstraction changes, retain the case• Stopping condition: failure of a few candidates

Page 8: Improving Test Suites via Operational Abstraction

Michael Ernst, page 8

The operational difference technique is

effectiveOperational difference suites

• are smaller• have better fault detection

than branch coverage suites

(in our evaluation; see paper for details)

Page 9: Improving Test Suites via Operational Abstraction

Michael Ernst, page 9

Example of test suite generation

Program under test: abs (absolute value)Test cases: 5, 1, 4, -1, 6, -3, 0, 7, -8, 3, …Suppose an operational abstraction contains:

• var = constant• var ≥ constant• var ≤ constant• var = var• property property

Page 10: Improving Test Suites via Operational Abstraction

Michael Ernst, page 10

Considering test case 5Initial test suite: { }Initial operational abstraction for { }: Ø Candidate test case: 5New operational abstraction for { 5 }:

• Precondition: arg = 5• Postconditions: arg = return

New operational abstraction is different,so retain the test case

Page 11: Improving Test Suites via Operational Abstraction

Michael Ernst, page 11

Considering test case 1Operational abstraction for { 5 }:

• Pre: arg = 5• Post: arg = return

Candidate test case: 1New operational abstraction for { 5, 1 }:

• Pre: arg ≥ 1• Post: arg = return

Retain the test case

Page 12: Improving Test Suites via Operational Abstraction

Michael Ernst, page 12

Considering test case 4Operational abstraction for { 5, 1 }:

• Pre: arg ≥ 1• Post: arg = return

Candidate test case: 4New operational abstraction for { 5, 1, 4 }:

• Pre: arg ≥ 1• Post: arg = return

Discard the test case

Page 13: Improving Test Suites via Operational Abstraction

Michael Ernst, page 13

Considering test case -1

Operational abstraction for { 5, 1 }:• Pre: arg ≥ 1• Post: arg = return

Candidate test case: -1New operational abstraction for { 5, 1, -1 }:

• Pre: arg ≥ -1• Post: arg ≥ 1 (arg = return) arg = -1 (arg = -return) return ≥ 1

Retain the test case

Page 14: Improving Test Suites via Operational Abstraction

Michael Ernst, page 14

Considering test case -6

Operational abstraction for { 5, 1, -1 }:• Pre: arg ≥ -1• Post: arg ≥ 1 (arg = return) arg = -1 (arg = -return) return ≥ 1

Candidate test case: -6New operational abstraction for { 5, 1, -1, -6 }:

• Pre: Ø• Post: arg ≥ 1 (arg = return) arg ≤ -1 (arg = -return) return ≥ 1

Retain the test case

Page 15: Improving Test Suites via Operational Abstraction

Michael Ernst, page 15

Considering test case -3

Operational abstraction for { 5, 1, -1, -6 }:• Post: arg ≥ 1 (arg = return) arg ≤ -1 (arg = -return) return ≥ 1

Test case: -3New operational abstraction for { 5, 1, -1, 6, -3 }:

• Post: arg ≥ 1 (arg = return) arg ≤ -1 (arg = -return) return ≥ 1

Discard the test case

Page 16: Improving Test Suites via Operational Abstraction

Michael Ernst, page 16

Considering test case 0Operational abstraction for { 5, 1, -1, -6 }:

• Post: arg ≥ 1 (arg = return) arg ≤ -1 (arg = -return) return ≥ 1

Test case: 0New operational abstraction for {5, 1, -1, -6, 0 }:

• Post: arg ≥ 0 (arg = return) arg ≤ 0 (arg = -return) return ≥ 0

Retain the test case

Page 17: Improving Test Suites via Operational Abstraction

Michael Ernst, page 17

Considering test case 7Operational abstraction for { 5, 1, -1, -6, 0 }:

• Post: arg ≥ 0 (arg = return) arg ≤ 0 (arg = -return) return ≥ 0

Candidate test case: 7New operational abstraction for { 5, 1, -1, -6, 0, 7 }:

• Post: arg ≥ 0 (arg = return) arg ≤ 0 (arg = -return) return ≥ 0

Discard the test case

Page 18: Improving Test Suites via Operational Abstraction

Michael Ernst, page 18

Considering test case -8

Operational abstraction for { 5, 1, -1, -6, 0 }:• Post: arg ≥ 0 (arg = return) arg ≤ 0 (arg = -return) return ≥ 0

Candidate test case: -8New operational abstraction for { 5, 1, -1, -6, 0, -8 }:

• Post: arg ≥ 0 (arg = return) arg ≤ 0 (arg = -return) return ≥ 0

Discard the test case

Page 19: Improving Test Suites via Operational Abstraction

Michael Ernst, page 19

Considering test case 3Operational abstraction for { 5, 1, -1, -6, 0 }:

• Post: arg ≥ 0 (arg = return) arg ≤ 0 (arg = -return) return ≥ 0

Candidate test case: 3New operational abstraction for { 5, 1, -1, -6, 0, 3 }:

• Post: arg ≥ 0 (arg = return) arg ≤ 0 (arg = -return) return ≥ 0

Discard the test case; third consecutive failure

Page 20: Improving Test Suites via Operational Abstraction

Michael Ernst, page 20

Minimizing test suitesGiven: a test suiteFor each test case in the suite:

Remove the test case if doing so does not change the operational abstraction

Page 21: Improving Test Suites via Operational Abstraction

Michael Ernst, page 21

OutlineOperational difference technique for selecting

test casesGenerating operational abstractionsStacking and area techniques for comparing

test suitesEvaluation of operational difference techniqueConclusion

Page 22: Improving Test Suites via Operational Abstraction

Michael Ernst, page 22

Dynamic invariant detection

Goal: recover invariants from programsTechnique: run the program, examine valuesArtifact: Daikon

http://pag.lcs.mit.edu/daikon

Experiments demonstrate accuracy, usefulness

Page 23: Improving Test Suites via Operational Abstraction

Michael Ernst, page 23

Goal: recover invariants

Detect invariants (as in asserts or specifications)• x > abs(y)• x = 16*y + 4*z + 3• array a contains no duplicates• for each node n, n = n.child.parent• graph g is acyclic• if ptr null then *ptr > i

Page 24: Improving Test Suites via Operational Abstraction

Michael Ernst, page 24

Uses for invariants• Write better programs [Gries 81, Liskov 86]

• Document code• Check assumptions: convert to assert• Maintain invariants to avoid introducing bugs• Locate unusual conditions • Validate test suite: value coverage• Provide hints for higher-level profile-directed compilation

[Calder 98]

• Bootstrap proofs [Wegbreit 74, Bensalem 96]

Page 25: Improving Test Suites via Operational Abstraction

Michael Ernst, page 25

Ways to obtain invariants

• Programmer-supplied• Static analysis: examine the program text

[Cousot 77, Gannod 96]• properties are guaranteed to be true• pointers are intractable in practice

• Dynamic analysis: run the program• complementary to static techniques

Page 26: Improving Test Suites via Operational Abstraction

Michael Ernst, page 26

Dynamic invariant detection

Look for patterns in values the program computes:• Instrument the program to write data trace files• Run the program on a test suite• Invariant engine reads data traces, generates potential

invariants, and checks them

Invariants

Instrumentedprogram

Originalprogram

Test suite

RunInstrument

Data tracedatabase

Detectinvariants

Page 27: Improving Test Suites via Operational Abstraction

Michael Ernst, page 27

Checking invariantsFor each potential invariant:

• instantiate(determine constants like a and b in y = ax + b)

• check for each set of variable values• stop checking when falsified

This is inexpensive: many invariants, each cheap

Page 28: Improving Test Suites via Operational Abstraction

Michael Ernst, page 28

Improving invariant detection

Add desired invariants: implicit values, unused polymorphism

Eliminate undesired invariants: unjustified properties, redundant invariants, incomparable variables

Traverse recursive data structuresConditionals: compute invariants over subsets

of data (if x>0 then yz)

Page 29: Improving Test Suites via Operational Abstraction

Michael Ernst, page 29

OutlineOperational difference technique for selecting

test casesGenerating operational abstractionsStacking and area techniques for comparing

test suitesEvaluation of operational difference techniqueConclusion

Page 30: Improving Test Suites via Operational Abstraction

Michael Ernst, page 30

Comparing test suitesKey metric: fault detection

• percentage of faults detected by a test suite

Correlated metric: test suite size• number of test cases• run time

Test suite comparisons must control for size

Page 31: Improving Test Suites via Operational Abstraction

Michael Ernst, page 31

Test suite efficiencyEfficiency = (fault detection)/(test suite size)

Which test suite generation technique is better?

faul

t det

ectio

n

test suite size

S2S1

Page 32: Improving Test Suites via Operational Abstraction

Michael Ernst, page 32

Different size suites are incomparable

A technique induces a curve:

How can we tell which is the true curve?

Page 33: Improving Test Suites via Operational Abstraction

Michael Ernst, page 33

Comparing test suite generation techiques

Each technique induces a curveCompare the curves, not specific pointsApproach: compare area under the curve

• Compares the techniques at many sizes• Cannot predict the size users will want

Page 34: Improving Test Suites via Operational Abstraction

Michael Ernst, page 34

Approximating the curves (“stacking”)

Given a test budget (in suite execution time),generate a suite that runs for that long

To reduce in size:select a random subset

To increase in size:combine independent suites

T1

T2S2

S2'

S1

S1'

faul

t det

ectio

n

size

Page 35: Improving Test Suites via Operational Abstraction

Michael Ernst, page 35

Test suite generation comparison

1. Approximate the curves2. Report ratios of

areas under curves T1

T2S2

S2'

S1

S1'

faul

t det

ectio

n

size

Page 36: Improving Test Suites via Operational Abstraction

Michael Ernst, page 36

OutlineOperational difference technique for selecting

test casesGenerating operational abstractionsStacking and area techniques for comparing

test suitesEvaluation of operational difference techniqueConclusion

Page 37: Improving Test Suites via Operational Abstraction

Michael Ernst, page 37

Evaluation of operational difference

technique• It ought to work: Correlating operational

abstractions with fault detection• It does work: Measurements of fault

detection of generated suites

Page 38: Improving Test Suites via Operational Abstraction

Michael Ernst, page 38

Subject programs8 C programs

• seven 300-line programs, one 6000-line program

Each program comes with• pool of test cases (1052 – 13585)• faulty versions (7 – 34)• statement, branch, and def-use coverage suites

Page 39: Improving Test Suites via Operational Abstraction

Michael Ernst, page 39

Improving operational abstractions improves

testsLet the ideal operational abstraction be that

generated by all available test casesOperational coverage = closeness to the ideal

• Operational coverage is correlated with fault detection• Holding constant cases, calls, statement coverage,

branch coverage• Same result for 100% statement/branch coverage

Page 40: Improving Test Suites via Operational Abstraction

Michael Ernst, page 40

Generated suitesRelative fault detection (adjusted by using the

stacking technique):Def-use coverage: 1.73Branch coverage: 1.66Operational difference: 1.64Statement coverage: 1.53Random: 1.00

Similar results for augmentation, minimization

Page 41: Improving Test Suites via Operational Abstraction

Michael Ernst, page 41

AugmentationRelative fault detection (via area technique):

Random: 1.00Branch coverage: 1.70Operational difference: 1.72Branch + operational diff.: 2.16

Page 42: Improving Test Suites via Operational Abstraction

Michael Ernst, page 42

Operational difference complements structural

Best techniqueTotal

Op. Diff. equal BranchCFG changes 9 11 9 29

Non-CFG changes 56 54 24 134

Total 65 65 33 163

Page 43: Improving Test Suites via Operational Abstraction

Michael Ernst, page 43

OutlineOperational difference technique for selecting

test casesGenerating operational abstractionsStacking and area techniques for comparing

test suitesEvaluation of operational difference techniqueConclusion

Page 44: Improving Test Suites via Operational Abstraction

Michael Ernst, page 44

Future workHow good is the stacking approximation?How do bugs in the programs affect the

operational difference technique?

Page 45: Improving Test Suites via Operational Abstraction

Michael Ernst, page 45

ContributionsStacking and area techniques for comparing test

suites• Control for test suite size

Operational difference technique for automatic test case selection• Based on observed program behavior• Outperforms statement and branch coverage• Complementary to structural techniques• Works even at 100% code coverage• No oracle, static analysis, or specification required