parameterized unit testing ictss’11 tutorial nikolai tillmann microsoft research
TRANSCRIPT
Outline
Unit TestingIsolationParameterized Unit Testing
Data Generation byDynamic Symbolic Execution
PatternsLimitations and other Details
About the exercises…
Interactive: http://pexforfun.com
Demos: Pex for Visual Studiohttp://research.microsoft.com/pex Requires Windows, .NET 2.0 or 4.0, ideally Visual Studio 2008 / 2010
Unit Testing
void AddAndCount() { var list = new List(); list.Add(3); Assert.AreEqual(1, list.Count);}
A unit test is a small program with assertions
Test a single (small) unit of code
Let’s play a game!
The Code Under Test
string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); foreach (var line in lines) { int index = line.IndexOf('='); string name = line.Substring(index); if (name == "Foo") { string value = line.Substring(index + 1); return value; } } return null;}
t:\myapp.iniA=BFoo=CC=D
Quiz: Coverage
How much block coverage do we need?1. 50%2. 80%3. 100%4. Block coverage alone is not enough
Quiz: Coverage
How much block coverage do we need?1. 50%2. 80%3. 100%4. Block coverage alone is not
enough
Quiz: Coverage
Do we need more tests to get 100% cov.?
[TestMethod]void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[0]); Reader.ReadFooValue();}
[TestMethod]void ExistingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“Foo=b”}); Reader.ReadFooValue();}
Quiz: Assertions
Why write Assertions (or not)?1. Documentation2. Double check your program3. Please your manager4. Prevent future bugs5. Validate user inputs6. Catch errors early
Quiz: Assertions
Why write Assertions (or not)?1. Documentation2. Double check your program3. Please your manager4. Prevent future bugs5. Validate user inputs6. Catch errors early
Quiz: Assertions
Which Assertions should you write?1. Assert.IsTrue(value == “b”);2. Assert.IsTrue(value == null);3. Assert.IsTrue(String.IsNullOrEmpty(value
))4. Assert.IsTrue(false);5. No assertions
[TestMethod]void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“a=b”}); string value = Reader.ReadFooValue(); Assert.IsTrue(????);}
Quiz: Assertions
Which Assertions should you write?1. Assert.IsTrue(value == “b”);2. Assert.IsTrue(value == null);3. Assert.IsTrue(String.IsNullOrEmpty(value
))4. Assert.IsTrue(false);5. No assertions
[TestMethod]void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“a=b”}); string value = Reader.ReadFooValue(); Assert.IsTrue(????);}
Quiz: Coverage + Assertions
What gives you confidence in the code?1. High coverage, no assertions2. Low coverage, many assertions3. High coverage, many assertions4. Low coverage, no assertions5. I wrote it
Quiz: Coverage + Assertions
What gives you confidence in the code?1. High coverage, few assertions2. Low coverage, many assertions3. High coverage, many assertions4. Low coverage, no assertions5. I wrote it
The Code Under Test
string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); foreach (var line in lines) { int index = line.IndexOf('='); string name = line.Substring(index); if (name == "Foo") { string value = line.Substring(index + 1); return value; } } return null;}
Quiz: Isolation
In the example, what are the external dependencies?1. Network Share2. Local Disk3. No file system, all in memory
string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); ...}
Quiz: Isolation
In the example, what are the external dependencies?1. Network Share2. Local Disk3. No file system, all in memory
string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); ...}
Quiz: Isolation
What is the problem with a Local Disk?1. Mapping already exists2. Cannot run tests concurrently3. Disk full4. Access rights
Quiz: Isolation
What is the problem with a Local Disk?1. Mapping already exists2. Cannot run tests concurrently3. Disk full4. Access rights
Quiz: Isolation
How do you mitigate the Local Disk issues?1. Always run on the same machine, same
hardware, same credentials, same time, same temperature, same solar system configuration
2. Refactoring: use Streams3. Refactoring: introduce IFileSystem4. Refactoring: pass the lines as parameter5. Change implementation of
File.ReadAllLines
Quiz: Isolation
How do you mitigate the Local Disk issues?1. Always run on the same machine, same
hardware, same credentials, same time, same temperature, same solar system configuration
2. Refactoring: use Streams3. Refactoring: introduce IFileSystem4. Refactoring: pass the lines as parameter5. Change implementation of
File.ReadAllLines
Quiz: Definition of Unit Test
What is a Unit Test? A Unit Test is a program that
runs fast the code under test, without environment dependencies, with assertions.
What is a Unit Test Suite? A set of Unit Tests which achieves high
code coverage
How do you change File.ReadAllLines?1. Override static method2. Changing the .NET Framework source
code(and recompiling it)
3. Rewrite application in JavaScript4. Code instrumentation
Dependency hell
var lines = File.ReadAllLines(@"t:\v.ini");
Replace any .NET methodwith a delegate
var lines = File.ReadAllLines(@"t:\myapp.ini");
File.ReadAllLines = delegate(string fn) MFile.ReadAllLinesString = delegate(string fn){ return new string[0];};
What if we could replace File.ReadAllLines?
Moles
Motivation for Moles
Why another isolation framework? Specifically designed to enable Pex
Simple, well-defined semantics▪ “Replace any .NET method”
Mole Types Code Generation
// System.IOpublic static class File { public static string[] ReadAllLines(string fn);}
// System.IO.Molespublic class MFile { public static Func<string, string[]> ReadAllLinesString;}// delegate R Func<T, R>(T t);
Injecting Detours at Runtime
// System.IOpublic static class File { public static string[] ReadAllLines(string fn) { if (MFile.ReadAllLinesString != null) return MFile.ReadAllLines(fn); … original code }}
Automatically injected
at runtime
MolesDemo
Start from ReadFooValue problem Create test project Add moles for mscorlib Write test using moles Run test Debug test
You can pick up project fromReadFooValueIsolation\ReadFooValueIsolation.sln
Quiz: Moles Usage
When should you use Moles (and not)?1. Always use Moles to solve isolation issues2. With Moles, one does not need to use interfaces
anymore3. Moles only should be used for 3rd party API, use
interfaces for isolation in your APIs4. Moles can be used in production code5. Moles lets you get away with untestable APIs6. Moles make test cases more robust7. With Moles, you do not need integration tests anymore8. Moles make tests easier to understand9. Moles is for poor programmers, real programmers rely
on interfaces
Quiz: Moles Usage
When should you use Moles (and not)?1. Always use Moles to solve isolation issues2. With Moles, one does not need to use interfaces
anymore3. Moles only should be used for 3rd party API, use
interfaces for isolation in your APIs4. Moles can be used in production code5. Moles lets you get away with untestable APIs6. Moles make test cases more robust7. With Moles, you do not need integration tests anymore8. Moles make tests easier to understand9. Moles is for poor programmers, real programmers rely
on interfaces
Exercise (Optional / homework)
Step-by-step tutorial: research.microsoft.com/documentati
on.aspx “Unit Testing with Microsoft Moles” “Unit Testing SharePoint Foundation
with Microsoft Pex and Moles”
What you learned so far
The Definition of Unit Testing
Unit Test Isolation
1. Identify External Dependencies2. Replace each External API with your
delegate
The Recipe of Unit Testing
var list = new List(); list.Add(item); var count = list.Count;
Assert.AreEqual(1, count);}
Three essential ingredients: Data Method Sequence Assertionsvoid AddAndCount() { int item = 3;
Quiz: list.Add(???)
Which value matters?1. 02. 13. int.MaxValue, int.MinValue4. -15. 10006. it does not matter7. I don’t know until I read the code
list.Add(???);
Parameterized Unit Test = Unit Test with Parameters
Separation of concerns Specification of behavior Data to achieve coverage
Parameterized Unit Testingvoid AddAndCount(List list, int item) { var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}
for any list,
for any item,
… adding 1 item increments Count
by 1
Parameterized Unit Tests areAlgebraic Specifications
A Parameterized Unit Test can be read as a universally quantified, conditional axiom.void ReadWrite(string name, string data) { Assume.IsTrue(name != null && data != null); Write(name, data); var readData = Read(name); Assert.AreEqual(data, readData);}forall. string name, string data: name not_equal null and data not_equal null implies equals( ReadResource(name,WriteResource(name,data)), data)
Parameterized Unit Testingis going mainstreamParameterized Unit Tests (PUTs) commonly
supported by various test frameworks .NET: Supported by .NET test frameworks
http://www.mbunit.com/ http://www.nunit.org/ …
Java: Supported by JUnit 4.X http://www.junit.org/
Generating test inputs for PUTs supported by tools .NET: Supported by Microsoft Research Pex
http://research.microsoft.com/Pex/ Java: Supported by Agitar AgitarOne
http://www.agitar.com/
Problem
We cannot execute parameterized unit tests without data.
Where does the data come from?
Random data generator Real customer data Ranges Some values hand-picked by
dev/tester
45
Goal: Given a program with a set of input parameters, automatically generate a set of input values that, upon execution, will exercise as many statements as possible
How would you do it?
Data Generation Challenge
Combines ideas from symbolic execution
to capture behavior symbolically model checking
to perform systematic bounded state space exploration
testing to monitor and check concrete behavior
constraint solving to find inputs
Dynamic Symbolic ExecutionCombines many ideas
47
Combines concrete and symbolic execution
Algorithm:Set J := ∅ (J is set of already analyzed
program inputs)Loop
Choose program input i ∉ J (stop if no such i can be found)Output iExecute P(i); record path condition C (in particular, C(i) holds)Set J := J ∪ C (viewing C as the set { i | C(i ) } )
End loop
Loop does not terminate if number of execution paths is infinite (in the presence of loops/recursion)
Dynamic Symbolic ExecutionThe high-level algorithm• This choice decides search
order• Search order decides how quick we can achieve high code coverage!• Incomplete constraint-solver leads to under-approximation
Code to generate inputs for:
Constraints to solve
a!=null a!=null &&a.Length>0
a!=null &&a.Length>0 &&a[0]==1234567890
void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
Observed constraints
a==nulla!=null &&!(a.Length>0)a!=null &&a.Length>0 &&a[0]!=1234567890
a!=null &&a.Length>0 &&a[0]==1234567890
Data
null
{}
{0}
{123…}a==null
a.Length>0
a[0]==123…T
TF
T
F
F
Execute&MonitorSolve
Choose next path
Done: There is no path left.
Dynamic Symbolic ExecutionExample
Dynamic Symbolic Execution Exercises
ArrayIndexLengthPex knows about all implicit, exception-throwing control-flow branches
ArrayHeapPex models the heap
Assert, Assert123Assertions connect code coverage and correctnesshttp://pex4fun.com/DynamicSymbolicExecutionExercises
Assertions vs. Coverage
What about finding faults?
void AssertIsTrue(bool condition) { if (!condition) throw new AssertionException();}
Assertions induce branches Pex tries to cover branches▪ Pex tries to fail assertions -> finds
bug!
Create new project in Visual Studio. Insert CoverMe method below. Right-click on CoverMe method, and select “Run Pex”. Inspect results in table. Save tests, …, have fun!public static void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
Dynamic Symbolic Executionwith Pex in Visual Studio
Dynamic Symbolic Executionwith Pex in Visual Studio
Generated Test Inputs
are persisted
as C# Unit Tests
Exercise (Optional / homework)
Step-by-step tutorial: research.microsoft.com/documentati
on.aspx “Exploring Code with Microsoft Pex”
How to test this code?(Actual code from .NET base class libraries)
Motivation: Unit Testing HellResourceReader
DemoResourceReader
[PexClass, TestClass][PexAllowedException(typeof(ArgumentNullException))][PexAllowedException(typeof(ArgumentException))][PexAllowedException(typeof(FormatException))][PexAllowedException(typeof(BadImageFormatException))][PexAllowedException(typeof(IOException))][PexAllowedException(typeof(NotSupportedException))]public partial class ResourceReaderTest { [PexMethod] public unsafe void ReadEntries(byte[] data) { PexAssume.IsTrue(data != null); fixed (byte* p = data) using (var stream = new UnmanagedMemoryStream(p, data.Length)) { var reader = new ResourceReader(stream); foreach (var entry in reader) { /* reading entries */ } } }}
Defined by execution environment / programming language, symbolic execution precision, and constraint solving Execution environment: C, Java, x86, .NET,… Precision: linear vs. non-linear arithmetic, “gods integers” vs.
bitvectors, concrete heap vs. symbolic heap., floating-point values, etc.
Solvers: lp_solve, CVCLite, STP, Disolver, Z3,… Examples of DSE implementations:
DART (Bell Labs), and also CUTE “concolic execution” EXE/EGT/KLEE (Stanford) “constraint-based execution” Vigilante (Microsoft) to generate worm filters BitScope (CMU/Berkeley) for malware analysis Sage (Microsoft) for security testing of X86 code Yogi (Microsoft) to verify device drivers (integrated in SLAM) Pex (Microsoft) for parameterized unit testing of .NET code CREST, jCUTE, jFuzz, …
Dynamic Symbolic ExecutionMany implementations
Constraint Solvingvs. Search Strategies
TestInputs
Constraint System
Execution Path
KnownPaths
Initially, choose ArbitraryPath-constraint
solving is just hard.
Reachability is
undecidable! (Loops)
Representation of symbolic values and state is similar to the ones used to build verification conditions in ESC/Java, Spec#, …Terms for Primitive types (integers, floats, …), constants, expressions Struct types by tuples Instance fields of classes by mutable ”mapping of references to
values" Elements of arrays, memory accessed through unsafe pointers
by mutable “mapping of integers to values"
Efficiency by Many reduction rules, including reduction of ground terms to
constants Sharing of syntactically equal sub-terms BDDs over if-then-else terms to represent logical
operations Patricia Trees to represent AC1 operators
(including parallel array updates)
Symbolic State Representation
SMT-Solver (“Satisfiability Modulo Theories”) Decides logical first order formulas with respect to
theories SAT solver for Boolean structure Decision procedures for relevant theories:
uninterpreted functions with equalities,linear integer arithmetic, bitvector arithmetic, arrays, tuples
Model generation for satisfiable formulas Models used as test inputs
Limitations No decision procedure for floating point arithmetic
Pex uses Z3: http://research.microsoft.com/z3
Constraint Solving
)1()2),3,,(((2 xyfyxawritereadfyx
ArithmeticArray TheoryUninterpreted
Functions
Satisfiability Modulo Theories
void Foo(int x, int y, int[] a) { if (x + 2 == y) { a[x] = 3; Debug.Assert(f(a[y - 2]) == f(y – x + 1)); }}
Pex uses some heuristic approaches for domain-specific problems.
Strings [TACAS’09, ICST’10] “uninterpreted functions” for string operations two-phase solving: first solve integer constraints
involving indices and lengths, then character constraints
Floating-point arithmetic Search-based approach: minimization of fitness
function … (more to come)
Domain-specific Constraint Solving
string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); foreach (var line in lines) { int index = line.IndexOf('='); string name = line.Substring(index); if (name == "Foo") { string value = line.Substring(index + 1); return value; } } return null;}
ExerciseWrite PUT for ReadFooValue
Can you come up with Parameterized Unit Tests?
Create new solution with C# class library Write code for ReadFooValue Create Test Project Add moles for mscorlib Write parameterized unit tests.
Or get result from ReadFooValuePUT\ReadFooValuePUT.sln
ExerciseWrite PUT for ReadFooValue
[PexMethod] // attribute marks pex testspublic void Test(...) { ...}
ExerciseCrash Test
[PexMethod]void Crash(string[] lines) { MFile.ReadAllLines = () => lines; Reader.ReadFooValue();}
Execute the code without assertions
ExerciseAssert Observed Results
[PexMethod]public string Regression(string[] lines) { MFile.ReadAllLines = () => lines; return Reader.ReadFooValue();}
Manually review outputs Pex inserts Assertions automatically
Exercise
[PexMethod]void FooExist(string value1) { PexAssume.IsTrue(value1 != null); var lines = new string[]{“Foo=“ + value1}; MFile.ReadAllLines = () => lines; var value2 = Reader.ReadFooValue();
PexAssert.AreEqual(value1, value2);}
Write and read property
Test Project
Test Generation Work-Flow
Code-Under-Test
Project
Parameterized Unit Tests
Pex Generated Tests
[PexMethod]public void ArrayListTest(ArrayList al, object o) { PexAssume.IsTrue(al != null); int len = al.Count; al.Add(o); PexAssert.IsTrue(al[len] == o);}
What are the test inputs? Only null An array list created by calling a constructor ... and also calling Add(…) many times … and also calling Clear() many times …
Quiz:Complex Objects As Inputs
[PexMethod]public void ArrayListTest(ArrayList al, object o) { PexAssume.IsTrue(al != null); int len = al.Count; al.Add(o); PexAssert.IsTrue(al[len] == o);}
Other possible problems: What if… ArrayList has no public constructor ArrayList is an abstract class (or
interface)
Quiz:Complex Objects As Inputs
Problem: Test parameters may be classes Symbolic execution tracks constraints over
(private) fields of objects Constraint solver determines solution,
assigning concrete values to (private) fields However, in practice objects must be
initialized by constructor, and can be mutated only by calling certain methods
What sequence of method calls reaches a desired field assignment? There may be no such sequence In general, undecidable
Creating complex objectsThe Problem
Factory methods Parameterized factories create and configure
objects Explicit: Public static method annotated with
[PexFactoryMethod] Implicit: Pex guesses “best” constructor/setters
Result: Exploration of reachable states Reachable using factory method Reachable within configured bounds Under-approximation of possible states
Object creation:Factory methods
Hand-written factory method:
[PexFactoryMethod(ArrayList)]public static ArrayList Create( int capacity, object[] values) { var a = new ArrayList(capacity); foreach (value in values) a.Add(value); return a;}
Pex will explore both factory method and PUT!
Factory methodsArrayList
Open project “ArrayListSample”. Explore AddTestExplicit Explore AddTestImplicit
Inspect generated test cases Notice constructor implicitly used by Pex Save factory method Edit factory method Explore again
Demo:Factory methods
Write class invariant as boolean-valued parameterless method Refers to private fields Must be placed in implementation code
Write special constructor/factory method for testing only Sets fields, and assumes invariant
Result: Exploration of feasible states May include states that are not reachable
AlternativeObject creation via class invariants
Pattern4A
Assume, Arrange, Act, Assert
[PexMethod]void Add(List target, T value) { PexAssume.IsNotNull(target); // assume var count = target.Count; // arrange target.Add(value); // act
// assert quiz1. Assert(target != null); 2. Assert(target.Count == count + 1); 3. Assert(target[0] == value); }
Pattern4A
Assume, Arrange, Act, Assert
[PexMethod]void Add(List target, T value) { PexAssume.IsNotNull(target); // assume var count = target.Count; // arrange target.Add(value); // act
// assert quiz1. Assert(target != null); 2. Assert(target.Count == count + 1); 3. Assert(target[0] == value); }
PatternRegression Tests
Generated test asserts any observed value Return value, out parameters
When code evolves, breaking changes in observable will be discovered
int AddTest(int a, int b) { return a + b; }
void AddTest01() { var result = AddTest(0, 0); Assert.AreEqual(0, result);}
PatternRoundtrip
For an API f(x), f-1(f(x)) = x for all x
void ToStringParseRoundtrip(int value) { string s = value.ToString(); int parsed = int.Parse(s);
// assert quiz1. Assert.AreEqual(value, s);2. Assert.AreEqual(s, parsed);3. Assert.AreEqual(parsed, value);}
PatternRoundtrip
For an API f(x), f-1(f(x)) = x for all x
void ToStringParseRoundtrip(int value) { string s = value.ToString(); int parsed = int.Parse(s);
// assert quiz1. Assert.AreEqual(value, s);2. Assert.AreEqual(s, parsed);3. Assert.AreEqual(parsed, value);}
PatternNormalized Roundtrip
For an API f(x), f-1(f(f-1(x)) = f-1(x) for all x
void ParseToString(string x) { var normalized = int.Parse(x); var intermediate = normalized.ToString(); var roundtripped = int.Parse(intermediate);
// assert quiz1. Assert(x == intermediate);2. Assert(intermediate == roundtripped);3. Assert(normalized == roundtripped);4. Assert(x == roundtripped); }
PatternNormalized Roundtrip
For an API f(x), f-1(f(f-1(x)) = f-1(x) for all x
void ParseToString(string x) { var normalized = int.Parse(x); var intermediate = normalized.ToString(); var roundtripped = int.Parse(intermediate);
// assert quiz1. Assert(x == intermediate);2. Assert(intermediate == roundtripped);3. Assert(normalized == roundtripped);4. Assert(x == roundtripped); }
PatternReachability
Indicate which portions of a PUT should be reachable.
[PexAssertReachEventually]public void Constructor(object input){ new Foo(input); PexAssert.ReachEventually(); }
PatternSame Observable Behavior
Given two methods f(x) and g(x), and a method b(y) that observes the result or the exception behavior of a method, assert that f(x) and g(x) have same observable behavior under b, i.e. b(f(x)) = b(g(x)) for all x.
public void ConcatsBehaveTheSame( string left, string right) { PexAssert.AreBehaviorsEqual( () => StringFormatter.ConcatV1(left, right), () => StringFormatter.ConcatV2(left, right));}
PatternCommutative Diagram Given two implementations f and g of the
same function, each possible requiring a different number of steps, i.e. f(x)=f1(f2(…(fn(x)…)), and g(x)=g1(g2(… (gm(x)…)), then it should hold thatf1(f2(…(fn(x))…) = g1(g2(…(gm(x)…)) for all x.
string Multiply(string x, string y);int Multiply(int x, int y);
void CommutativeDiagram1(int x, int y) { string z1 = Multiply(x, y).ToString(); string z2 = Multiply(x.ToString(), y.ToString()); PexAssert.AreEqual(z1, z2);}
HomeworkPatterns
Read patterns paper: research.microsoft.com/Pex
/documentation.aspx “Parameterized Test Patterns for Microsoft
Pex”
Pex understand managed .NET code only Pex does not understand native code.
Problem if branching over values obtained from the environment Pex may not automatically detect all such
cases.
Testability
if (!File.Exists(f)) throw ...
File System?
Hidden Complexity
Pex analyzes every executed .NET instruction
Some used libraries may be surprisingly expensive to analyze XML parsing repeatedly converting data between different
representationsvoid Sum(string[] A) { var sum = “0”; foreach(var a in A) sum = (int.Parse(a) + int.Parse(sum)).ToString(); if(sum == “123”) throw new Exception(); }
Don’t do this.
Exploration Boundaries
Configurable bounds include: TimeOut MaxBranches MaxCalls MaxConditions▪ Number of conditions that depend on test
inputs MaxRuns ConstraintSolverTimeOut ConstraintSolverMemoryLimit
Multi-threaded code
Unlike test inputs, thread-interleavings can normally not be controlled
Thus, Pex can only explore single-threaded code
Related approach to explore thread-schedules (but not input parameters) by controlling thread-scheduler: CHESShttp://research.microsoft.com/CHESS
ConclusionWhat you learned today
The Definition of Unit Testing Unit Tests are not Integration Tests
Unit Test Isolation Introduction of Abstraction Layers
Parameterized Unit Testing Separation of concerns:
Coverage/Specification Patterns to write Parameterized Unit Tests Data Generation by Dynamic Symbolic
Execution
Availability
Pex and Moles areVisual Studio 2010 Power Tools
(academic/commercial license available)
Minimum: Windows XP, .NET 2.0 Ideally: Visual Studio 2010 Professional http://research.microsoft.com/pex/
http://pexforfun.com