Using Dynamic Using Dynamic Compilers for Software Compilers for Software

TestingTesting

Ben BreechBen BreechLori PollockLori Pollock

John CavazosJohn Cavazos

Background: Static CompilersBackground: Static Compilers

sourcecode

Lexical, syntactic, Semantic analysis

Inter. Rep.(basic blocks)

Static AnalysisOptimization

Code generationExecutable

Code

Static Compiler

Background: Background: Software TestingSoftware Testing

Assesses Assesses qualityquality of software of software• Correctness, performance, securityCorrectness, performance, security

Most testing performed by executing codeMost testing performed by executing code

ExecutableCode

TestInput

ExecuteProgram

ProgramResults

ExpectedResults

Check Results(oracle)

Motivating ExampleMotivating Exampleif ((sptr = malloc (size + 1)) == NULL) {if ((sptr = malloc (size + 1)) == NULL) {

findmem ();findmem ();if ((sptr = malloc (size + 1)) == NULL)if ((sptr = malloc (size + 1)) == NULL)

xlfail (“insufficient string space”); }xlfail (“insufficient string space”); }

How do I test this callsite?How do I test this callsite?

Make the machine run out of memory?Make the machine run out of memory?Flip the conditional, recompile, flip back?Flip the conditional, recompile, flip back?

Pretend it doesn’t exist during testing?Pretend it doesn’t exist during testing?

Generalizing the ProblemGeneralizing the Problem

Code to handle Code to handle uncommon situationsuncommon situations• Difficult to testDifficult to test• May need external environment event to triggerMay need external environment event to trigger

Examples:Examples:• Error handling codeError handling code• Testing program security mechanismsTesting program security mechanisms

ObservationObservation

Hard to reach code executes when program Hard to reach code executes when program thinksthinks something uncommon has occurred something uncommon has occurred

if ((sptr = malloc (size + 1)) == NULL) {if ((sptr = malloc (size + 1)) == NULL) { findmem ();findmem ();

xlfail (“insufficient string space”); }xlfail (“insufficient string space”); }if ((sptr = malloc (size + 1)) == NULL)if ((sptr = malloc (size + 1)) == NULL)

Could test Could test findmemfindmem() by simulating error() by simulating error• E.g., could add instructions to program so program E.g., could add instructions to program so program

believes believes mallocmalloc failed failed

RUGRAT ApproachRUGRAT Approach

Use Use Dynamic CompilersDynamic Compilers to generate to generate test cases for hard to reach code.test cases for hard to reach code.

AutomaticallyAutomatically add instructions to add instructions to program program during executionduring execution to simulate to simulate uncommon situation.uncommon situation.

Dynamic CompilersDynamic Compilers

Dynamic compilers perform compilation Dynamic compilers perform compilation tasks tasks duringduring program execution program execution

code

Analysis&

optimization

Create basic block

translate

Basicblock

Mod. Basicblock

Executeon CPU

Dynamic Compiler

RUGRAT ArchitectureRUGRAT Architecture

code

Analysis&

optimization

Create basic block

translate

Basicblock

Mod. Basicblock

Executeon CPU

Dynamic Compiler

Create basic block

Dynatest Generator

Testspec

Test OracleTest

Report

Dynatest GeneratorDynatest Generator

Decides when/where/how to add testDecides when/where/how to add test• Decision based on test specificationDecision based on test specification

Test spec examplesTest spec examples• ““test all test all mallocsmallocs in program” in program”• ““test 2nd test 2nd mallocmalloc call in function call in function foofoo””

ExampleExampleif ((sptr = malloc (size + 1)) == NULL) {if ((sptr = malloc (size + 1)) == NULL) {

findmem ();findmem ();

xlfail (“insufficient string space”); }xlfail (“insufficient string space”); }

if ((sptr = malloc (size + 1)) == NULL)if ((sptr = malloc (size + 1)) == NULL)

call malloc (code for malloc)movl <return val> sptrcmpl sptr, 0jnz L1call findmem….L1: …

Dynatest Generator

call malloc (code for malloc)movl 0, <return val>movl ENOMEM, errnomovl <return val> sptrcmpl sptr, 0jnz L1call findmem….L1: …

Good Times, Bad TimesGood Times, Bad Times

The Bad: Not a perfect simulationNot a perfect simulation

The Good: Adequate simulationAdequate simulation Can target system or appl callsCan target system or appl calls Saves quite a lot of tester effortSaves quite a lot of tester effort

Security Mechanism Testing:Security Mechanism Testing:Encrypting Function PointersEncrypting Function Pointers

Protects progs against func pointer attacksProtects progs against func pointer attacks Difficult to test (need vulnerable program and Difficult to test (need vulnerable program and

attack)attack) RUGRAT can simulate attack by adding RUGRAT can simulate attack by adding

instructionsinstructions• Very different from error handling code caseVery different from error handling code case

RUGRAT can be used for variety of RUGRAT can be used for variety of testing tasks.testing tasks.

Experiments SummaryExperiments Summary

Tested variety programs with RUGRATTested variety programs with RUGRAT 120+ error code handling callsites covered120+ error code handling callsites covered

• Both application and system callsBoth application and system calls Increased error code coverage ~ 50% over Increased error code coverage ~ 50% over

regular test casesregular test cases• Not all error code statements could be coveredNot all error code statements could be covered

• Different options, etcDifferent options, etc

Reasonable time overheadReasonable time overhead

Some related workSome related work

Holodeck [1], FIG [2] Holodeck [1], FIG [2] • Require tester provide alternative “stub” Require tester provide alternative “stub”

functions to do testingfunctions to do testing• Miss application callsMiss application calls

Dynamic branch switching [3]Dynamic branch switching [3]• Not originally intended for testing error codeNot originally intended for testing error code• Need to know which branch to changeNeed to know which branch to change• Far less accurate simulationFar less accurate simulation

[1] Thompson et al., SAC 2002[1] Thompson et al., SAC 2002[2] Broadwell et al., SHAMAN 2002[2] Broadwell et al., SHAMAN 2002[3] Zhang et al., ICSE 2006[3] Zhang et al., ICSE 2006

Conclusions and SummaryConclusions and Summary

Presented RUGRAT architecturePresented RUGRAT architecture• Can test hard to reach (and seldom tested) Can test hard to reach (and seldom tested)

code by using dynamic compilerscode by using dynamic compilers• Saves tester effortSaves tester effort

RUGRAT is a general toolRUGRAT is a general tool

RUGRAT ArchitectureRUGRAT Architecture

code

Basicblock

Mod. Basicblock

Executeon CPU

Dynamic Compiler

Create basic block

Dynatest Generator

Testspec

Test OracleTest

Report