A Comparison of Online and Dynamic

Impact Analysis Algorithms

Ben Breech

Mike Tegtmeyer

Lori Pollock

University of Delaware

Impact Analysis

• If I change function C, what other functions could be affected (impacted)?

• Results useful for regression testing and maintenance

A

mainB

C

D

E

G H

F

Our Contributions

• New online impact analysis algorithm

• Experimentally compare two dynamic and two online algorithms• Space• Time• Scalability• Effectiveness

Static Impact Analysis

• Perform analysis on source code• Call graph traversals• Slicing

• Obtain conservative results+ Accounts for all possible inputs and behaviors- Can give very large impact sets

SourceCode

Static Analyzer Impacts

Dynamic Impact Analysis

- Not conservative -- results depend on input+ Give impacts related to program use• Algorithms:

• CoverageImpact (Orso et al. FSE 03)• PathImpact (Law & Rothermel, ICSE 03)

ExecuteInstrumented

Program

InstrumentProgram

Post ExecAnalysis

Dynamic Data

ImpactsInput

Dynamic: CoverageImpact

• Calculate • Function coverage by executing program• Static forward slice from changed function

• Intersection gives impacts

A

mainB

C

D

E

G H

F

main

A

C

D

F

E

Slice functions: {Main, A, B, C, D, E, F}

Function Coverage: {Main, A, C, D, E, F, G}

Find Impact of E:

(Orso et al. FSE 03)

Dynamic: PathImpact

• Calculate function execution trace• Compress using SEQUITUR (can be done at execution)• Traverse compressed trace to find impacts

• Once called, function impacts all future functions• Returns cause impacts

Amain

B

C

D

E

G H

F

Trace: Main G r A C F C r r D r E r D r r r x

mainA

C

D

E

Find Impact of E:

(Law & Rothermel, ICSE 03)

Our Approach:Online Impact Analysis

Dynamic Compiler

Executing Program

Impact Analysis Impacts

Input

Use dynamic compiler to perform impact analysis as program runs

(Breech et al., ICSM 2004)

Dynamic Compilers

Allows for …• modifying program instructions during

execution• analysis during execution

Executable code

DynamoRIO

Analysis and/or

Optimization(user module)

InstructionExecution

ProgramResults

(Bruening et al., CGO 2003)

Online: Optimistic

Optimistic impacts propagate only along call paths• Impacts of f are functions on call stack at the same time as f+ May be useful for programs with infrequent global variable use+ Calculates all impacts during one execution- May miss some impacts

(Breech et al., ICSM 2004)

Amain

B

C

D

E

G H

F

Trace: Main G r A C F C r r D r E r D r r r x (not actually stored, only for demonstration)

Find Impact of E:

mainA

C E

Call stack: main G A C F D E C

Online: PathImpact_AllIn1

• Same heuristics as PathImpact• Execution impacts all future functions• Returns cause impacts

• Calculates all impacts during one execution• Uses matrix of bits to keep track of impacts

Amain

B

C

D

E

G H

F

Trace: Main G r A C F C r r D r E r D r r r x (not actually stored, only for demonstration)

Find Impact of E:

mainA

C

D

E

Experimental Study

• Gain insight into tradeoffs of each alg• Used several small to medium sized

applications from SPEC + ESA space• 1,000 - 131,000 LOC• 19 - 2,015 functions

• Research Questions• How do the algs scale in time + space?• How do resulting impact sets compare?• Is it useful to apply dynamic compilers?

Experimental Setup: CoverageImpact

DynamoRIO

P

Module toCompute coverage

FunctionCoverage

CodeSurfer(http://www.grammatech.com)

ICFG

Post Execution Analyzer(Approx slicing with reachability)

Impacts

Experimental Setup: PathImpact

P

Instrumentusing SUIF

P’ CallTrace

SEQUITUR

Post ExecutionAnalyzer

Impacts

execute

Experimental Setup: Optimistic

P

DynamoRIO

Module forOptimistic

Impacts

Experimental Setup: PathImpact_AllIn1

Created 2 versions to examine usefulness of using dynamic compilers

P’Execute + Pipe Trace Analyze

During Exec.

Impacts

PDynamoRIO

Module forPathImpact_AllIn1

PInstrumentusing SUIF

Results: Timing

• CoverageImpact • Least run time overhead (~ 3 minutes max)• Fast post execution analysis (~ few minutes)

• PathImpact• Very high overhead (> 2 hours, in some cases)

• Optimistic• Medium overhead (usually ~ minutes)• > 2 hours for heavily recursive program

• PathImpact_AllIn1• Version implemented with dynamic compiler

• Low to medium overhead (15 minutes max)• Ran successfully on all programs

• Overhead of pipe version too high in some cases

Results: Space

• CoverageImpact• ICFG took ~100K to ~8 MB

• PathImpact• Uncompressed trace ~2KB to > 15GB• Compressed trace ~400B to 1.2MB

• Could not be completed for all traces

• Optimistic, PathImpact_AllIn1• Negligible storage required

Results: Impact Sets

Examined 008.espresso results• Largest app that all algs ran (186 exec funcs)• PathImpact and PathImpact_AllIn1 gave

same results• CoverageImpact reported ~30 more functions per

impact set than PathImpact for 172 funcs• CoverageImpact reported smaller sets for 10

functions -- one function called through function pointer

• Optimistic reported much smaller sets (but may miss impacts)

Qualitative Summary

Coverage Impact

PathImpact

Optimistic

PI_AllIn1

Infrastructure Instr. code

ICFG

Instr. code

CompressDyn. Comp Dyn. Comp

Static Analysis

ICFG slicing

(none) (none) (none)

Analysis Time low high med to high med

Spatial Overhead

low to medlow (if

compress)low low

Summary

• Presented new online impact analysis algorithm (PathImpact_AllIn1)

• Experimentally compared online against dynamic algs• PathImpact_Allin1

• Scaled better than PathImpact• Gave better impact sets than CoverageImpact

• Optimistic• Scaled well (not as good as PathImpact_Allin1)• Gave different impact sets (may be useful in OOP)

Future Work

• Improve precision of online impact analysis

• Other uses of dynamic compilers for software engineering