parameterized unit testing with pex tutorial.pdf

Parameterized Unit Testing with Pex: Tutorial

Nikolai Tillmann, Jonathan de Halleux, and Wolfram Schulte

Microsoft ResearchOne Microsoft Way, Redmond WA 98052, USA

{nikolait,jhalleux,schulte}@microsoft.com

This hands-on tutorial teaches the principles of Parameterized Unit Testing in VisualStudio with Pex, an automatic test input generator.

A parameterized unit test (PUT) is simply a method that takes parameters, calls thecode under test, and states assertions. Given a PUT written in a .NET language, Pexautomatically produces a small test suite with high code and assertion coverage. More-over, when a generated test fails, Pex can often suggest a bug fix. To do so, Pex performsa systematic program analysis, similar to path bounded model-checking. Pex learns theprogram behavior by monitoring execution traces, and uses a constraint solver to pro-duce new test cases with different behavior. At Microsoft, this technique proved highlyeffective in testing even an extremely well-tested component.

From a specification, the developer (1) writes parameterized unit tests in C# to reflectthe specification, and (2) develops code that implements the specification. The tutorialoutlinea key aspects to make this methodology successful in practice, including how towrite mock objects, as well as the theoretical foundations on which Pex is built.

This document is separated into two main parts. The first part provides detailed walk-through exercises on unit testing in Section 2, the methodology of Parameterized UnitTesting in Section 3, the usage of the Pex tool in Section 4, and ways to deal withthe environment in Section 5. The second part is for the advanced reader. It provides abackground on white box testing techniques in Section 6, and discusses in detail vari-ous aspects of the Pex tool in Section 7. Section 8 gives an overview of related work.Finally, Section 9 concludes.

This tutorial refers to Pex version 0.18. The latest version of Pex can be obtainedfrom http://research.microsoft.com/Pex/.

1 Introduction

Unit tests are becoming increasingly popular. A recent survey at Microsoft indicatedthat 79% of developers use unit tests [325]. Unit tests are written to document customerrequirements, to reflect design decisions, to protect against changes, but also, as part ofthe testing process, to produce a test suite with high code coverage that gives confidencein the correctness of the tested code.

The growing adoption of unit testing is due to the popularity of methods like XP(extreme programming) [31], test-driven development (TDD) [30], and test execu-tion frameworks like JUnit [186], NUnit [322] or MbUnit [90]. XP does not say how

P. Borba et al. (Eds.): PSSE 2007, LNCS 6153, pp. 141202, 2010.c Springer-Verlag Berlin Heidelberg 2010

142 N. Tillmann, J. de Halleux, and W. Schulte

and which unit tests to write. Moreover, test execution frameworks automate only testexecution; they do not automate the task of creating unit tests. Writing unit tests byhand can be a laborious undertaking. In many projects at Microsoft there are more linesof code for the unit tests than for the implementation being tested. Are there ways toautomate the generation of good unit tests? We think that Parameterized Unit Testing isa possible answer, and this is the topic of the tutorial.

We describe how to design, implement and test software using the methodology ofParameterized Unit Testing [314,315], supported by the tool Pex [276,263]. Pex, anautomated test input generator, leverages dynamic [141] symbolic execution [194] totest whether the software under test agrees with the specification. As a result, softwaredevelopment becomes more productive and the software quality increases. Pex producesa small test suite with high code coverage from Parameterized Unit Tests.

In effect, we combine two kinds of testing introduced in Chapter 2: 1) Testing forfunctional properties: Just as unit tests, Parameterized Unit Tests usually serve as spec-ifications of functional properties. 2) Structural testing: We analyze such tests with dy-namic symbolic execution, a structural testing technique.

To facilitate unit testing, mock objects are often used to isolate the test from the envi-ronment. We extend this notion to parameterized mock objects, which can be viewed asa model of the environment. Writing such parameterized mock objects, and generatingtests with them, is in effect a form of model-based testing (see Chapter 3).

The effectiveness of a test suite, whether written by hand, or generated from param-eterized unit tests, can be measured with mutation testing (see Chapter 8).

Parameterized Unit Tests are algebraic specifications [38] written as code. Anothername for this concept is theories [291,293] in the JUnit test framework. They appear asrow tests in MbUnit [90], and under other names in various other unit test frameworks.While Pex is a tool that can generate test inputs for .NET code, many other research andindustrial tools [141,297,67,66,296,142] exist that can generate test inputs in a similarway for C code, Java code and x86 code.

We introduce the concepts and illustrate the techniques with some examples. Weassume deterministic, single-threaded applications.

An earlier and shorter version of this tutorial on Parameterized Unit Testing with Pexcan be found in [91]. More documentation can be found on the Pex website [276].

2 Unit Testing Today

A unit test is a self-contained program that checks an aspect of the implementationunder test. A unit is the smallest testable part of the program. One can partition eachunit test into three parts: (1) exemplary data, (2) a method sequence and (3) assertions.

exemplary data can be considered as the test input that is passed to the methods asargument values,

method sequence based on the data, the developer builds a scenario that usuallyinvolves several method calls to the code-under-test,

assertions encode the test oracle of a unit test. The test fails if any assertion fails oran exception is thrown but not caught. Many unit test frameworks have special sup-port for expected exceptions, which can often be annotated with custom attributes.

Parameterized Unit Testing with Pex: Tutorial 143

The program in Listing 5.1 is an example of a unit test that checks the interplayamong some operations of .NETs ArrayList class. The example is written in C#,omitting the class context, as we will do often for brevity.

1 p u b l i c void AddTest ( ) {2 / / exemplary da ta3 i n t c a p a c i t y = 1 ;4 o b j e c t e l e m e n t = n u l l ;5 / / method sequence6 A r r a y L i s t l i s t = new A r r a y L i s t ( c a p a c i t y ) ;7 l i s t . Add ( e l e m e n t ) ;8 / / a s s e r t i o n s9 A s s e r t . I s T r u e ( l i s t [ 0 ] == e l e m e n t ) ;

10 }Listing 5.1. A unit test for the ArrayList class

The AddTest method first sets up its state, by picking the values 1 and null forcapacity and value, respectively.

Then, the test method performs a sequence of method calls, starting by creatingan instance of the ArrayList class with the selected capacity. An array list is acontainer whose size may change dynamically. Internally, it uses a fixed-length array asbacking storage. The array lists capacity is the allocated length of its current backingstorage. Next, the test adds the element to the array list.

Finally, the test checks an assertion, making sure that the array list at position 0contains element. This is the test oracle.

There is often more than one way to partition a unit test into the three differentparts (exemplary data, method sequence, assertions). For example, Listing 5.2 is a verysimilar test, in which the input data consists of more complex objects, including anobject for the element, and the initial instance of the array list itself.

In the unit testing frameworks NUnit [259,322] and Visual Studio Unit Test in VisualStudio 2008 Professional [249], a parameterless method such as AddTest is decoratedwith a custom attribute like [TestMethod] to designate it as a unit test. The class inwhich unit tests are defined is decorated with an attribute like [TestClass]. Usually,each unit test explores a particular aspect of the behavior of the class-under-test.

2.1 Benefits of Unit Testing

Software developers (and sometimes testers) write unit tests for different purposes. Design and specification: developers translate their understanding of the specifica-

tion into unit tests and/or code. Developers following the test-driven developmentwrite the unit tests before code, and therefore use the unit tests to drive the design.Nonetheless, unit tests may be written in all phases of the software developmentprocess. Also, developers capture exemplary customer scenarios as unit tests.


1 p u b l i c void AddTest2 ( ) {2 / / exemplary da ta3 o b j e c t e l e m e n t = new o b j e c t ( ) ;4 A r r a y L i s t l i s t = new A r r a y L i s t ( 1 ) ;5 / / method sequence6 l i s t . Add ( e l e m e n t ) ;7 / / a s s e r t i o n s8 A s s e r t . I s T r u e ( l i s t [ 0 ] == e l e m e n t ) ;9 }

Listing 5.2. A typical unit test

Code coverage and regression testing: developers or testers may write unit tests toincrease their confidence in the correctness of code that they have already written. Itis well known that a test suite which achieves high code coverage and checks manyassertions is a good indicator of code quality. In this way, unit tests represent asafety net that developers can use when refactoring the code. Unit tests are usuallysmall tests that run fast and give a quick feedback on effects of code changes.Additionally, several tools exist to automatically execute a suite of unit tests oneach code change that is committed to the source code repository.

Short feedback loop: as mentioned above, unit tests are usually written by the de-velopers themselves before or after writing the product code. When a unit test failsand exposes a bug, the feedback loop to get the bug fixed is very short.

Documentation: the resulting unit tests are commonly considered as documentationof correct program behavior.

Therefore, writing unit tests is increasingly becoming an essential part of software de-velopment processes.

2.2 A Critique of Unit TestingUnit testing faces several challenges:

Quality of unit tests: the quality of the unit tests is mostly dependent on the timethe developer is willing to invest in them.

Amount of unit tests: writing more unit tests does not necessarily increase the codecoverage. Therefore, the size of the test suite is not an indicator of the code quality.

New code with old tests: while the unit tests are actively edited when the developer isimplementing the code, such tests are usually not updated later on (besides syntacticcorrections when APIs are refactored). Therefore, if the developer changes the codeimplementation, for example by adding more special cases in the code, but does notupdate the unit tests, he might introduce a number of new untested behaviors.

Hidden integration test: ideally, a unit test should test the code in isolation. Thismeans that all environment dependencies, e.g database, file I/O, must be hiddenbehind an abstraction layer. During testing, the abstraction layer provides a fakeimplementation, also referred as mocks. In practice, it is very easy to leak suchdependencies.


Despite these potential difficulties, continuous research efforts are addressing suchissues.

2.3 Measurement of Test Quality: Code Coverage and AssertionsWhat is a good test suite? When do you have enough unit tests to ensure a minimumlevel of quality? Those are hard questions that developers face.

Our experience within Microsoft and from the industry indicates that a test suite withhigh code coverage and high assertion density is a good indicator for code quality. Codecoverage alone is generally not enough to ensure a good quality of unit tests and shouldbe used with care. The lack of code coverage to the contrary clearly indicates a risk, asmany behaviors are untested.

A statement is covered when at least one unit test executes this statement. The codecoverage is then usually computed by executing the entire unit test suite and computingthe ratio of covered statements.

Different notions of code coverage exist [32], including the following: Basic Block Coverage: This coverage is based on basic block representation of the

programs control flow graph. A "basic block" is a sequence of instructions, herethe MSIL instructions of .NET, that has one entry point, one exit point, and nobranches within the block. It is commonly used in the industry.

Branch Coverage: This coverage is computed by analyzing the coverage of explicitarcs. An arc is a control transfer from one basic block to another in the programcontrol flow graph.

Implicit Branch Coverage: This is an extension of the arc coverage where all ex-plicit and implicit arcs are considered. Implicit arcs occur for exceptional behaviorof instructions, for example when accessing a field, the arc that throws a null deref-erence exception is implicit.

These notions are the ones that Pex employs, namely those that rely on the control flowcoverage.

2.4 Unit Testing in .Net

This section gives a quick introduction to unit testing in .NET. If you are familiar withthe techniques and tools, you might as well skip it.

Unit test frameworks. Several frameworks exist in .NET to help developers effectivelyauthor and run unit tests. Although each framework has its own particularities, they allprovide a core set of services:

a custom attribute based system for tagging methods as unit tests, automatic detection and execution of such unit tests, a runner with reporting capabilities. The runner might be a simple console applica-

tion or an integrated GUI.

In this document, we will use the Visual Studio Unit Test test framework that comeswith Visual Studio.


2.5 Exercises

Exercise 1. Getting started with Visual Studio 2008 Professional (or better) In thisexercise, we go through the steps to create a new test project, author unit tests and runthem. This section is targeted to users who are new to Visual Studio Unit Test projectsin Visual Studio.

Part 1: Creating a New Test Project1. Go to File|New|Project....

2. On the left pane, select Visual C#|Test, then select the Test Project item. Select alocation for the new project and click Ok.

The content of the window shown above might vary depending on your VisualStudio installation.

3. Delete the sample files that were generated by the project wizard (Authoring-Tests.txt, ManualTest1.mht, UnitTest1.cs). (You can right-click a filename, and select Delete in the context menu.)


Part 2: Creating a Passing Unit Test

1. Right-click on the project node and select Add|New Test.

2. In the main pane, select the Unit Test item, update the test name to HelloWorld-Test.cs and hit Ok.

3. Open HelloWorldTest.cs and clean the generated code to have an empty classdefinition. This is a test class, tagged with a [TestClass] attribute. Such a testclass is often called a test fixture.

u s i n g System ;u s i n g System . Text ;u s i n g System . C o l l e c t i o n s . G e n e r i c ;u s i n g M i c r o s o f t . V i s u a l S t u d i o . T e s t T o o l s . U n i t T e s t i n g ;

namespace T e s t P r o j e c t 1 {[ T e s t C l a s s ] / / t h i s c l a s s c o n t a i n s u n i t t e s t sp u b l i c c l a s s He l loW or ldT es t {}

}4. We start by adding a test that will pass. Add a new public instance method to the

HelloWorldTest class, tagged with the TestMethodAttribute, that writes "HelloWorld" to the console:


[ TestMethod ] / / t h i s i s a t e s tp u b l i c vo id P a s s i n g T e s t ( ) {

Conso l e . W r i t e L i n e ( " h e l l o wor ld " ) ;}

5. In the Test View window (Test|Windows|Test View), select the PassingTesttest, then click on Run Selected icon (upper left).

6. The test result window displays the status of the current run. Each selected test isrepresented as a row in the report. In this case, PassingTest succeeded and youcan review the details of that particular test by double-clicking on the row.

7. The test details view gives various metrics about the test run (including its duration)as well as the console output.

Part 3: Creating a Failing Unit Test

1. In the following, we add a test that fails. Add a new public instance method to theHelloWorldTest class that throws an exception.

[ TestMethod ]p u b l i c vo id F a i l i n g T e s t ( ) {

throw new I n v a l i d O p e r a t i o n E x c e p t i o n ( " boom " ) ;}

Open the Test View and execute both tests,


2. The test result window now contains two test results; one for PassingTest andone for FailingTest.

3. Go to the FailingTest method and hit F9 to set a debugger breakpoint.

4. To start debugging the failing tests, go to the test result view and click on the Debugoriginal tests menu item

5. The debugger will automatically stop on the breakpoint set previously. If you arenot familiar with the Visual Studio debugger, this is a good time to get some expe-rience. The yellow line shows the statement that will be executed next.

Part 4: Creating a Negative Unit Test

1. Visual Studio Unit Test supports a special attribute, ExpectedExceptionAttribute ,that specifies that the test must throw an exception of a particular type. You can useit to write unit tests that check that parameter validation code works properly.

[ TestMethod ][ E x p e c t e d E x c e p t i o n ( t y p e o f ( I n v a l i d O p e r a t i o n E x c e p t i o n ) ) ]p u b l i c vo id E x p e c t e d E x c e p t i o n T e s t ( ) {

throw new I n v a l i d O p e r a t i o n E x c e p t i o n ( " boom " ) ;}

After running the three tests, you can see that the ExpectedExceptionTest testwas marked as a passing test since it threw the expected exception,


Part 5: Enabling Code Coverage (This exercise requires Visual Studio 2008 Test Edi-tion, or better. Visual Studio 2008 Professional is not sufficient.)

1. Visual Studio Unit Test in Visual Studio comes with a built-in code coverage sup-port. To enable this feature, go to Test|Edit Test Run Configurations|Local TestRun.

2. On the left pane, select Code Coverage then select the TestProject1.dll as-sembly to be instrumented and click Close. Execute PassingTest and Failing-Test, leaving aside ExpectedExceptionTest.


3. In the Test Results window, click on the code coverage icon (last icon on the right).

4. In the Code Coverage Results window, enable source code coloring.

5. Covered code is colored in light blue, while uncovered code is colored in red. In thisexample, we did not run the ExpectedExceptionTest. That is why this methodis colored in red.

To summarize, in this exercise, we learned how to create a new Visual Studio Unit Test test project in Visual Studio, how to author, execute and debug unit tests, how to enable code coverage and analyze the results.

Exercise 2. Unit testing the Luhn Algorithm using Test Driven Development In thisexercise, we will implement the Luhn validation algorithm using a test driven develop-ment (TDD) approach [30].

The TDD cycle consists of the following short steps:1. Add a test,2. Run it and watch it fail,3. Change the code as little as possible such that the test should pass,4. Run the test again and see it succeed,5. Refactor the code if needed.

We apply the TDD steps in the following.


Part 1: Credit Card Number Validation Specification Most credit card companies usea check digit encoding scheme [344]. A check digit is added to the original creditcard number, at the beginning or the end, and is used to validate the authenticity of thenumber. The most popular encoding algorithm is the Luhn algorithm [345] which canbe computed by the following steps:

1. Double the value of alternate digits of the primary account number beginning withthe second digit from the right (the first righthand digit is the check digit.)

2. Add the individual digits comprising the products obtained in Step 1 to each of theunaffected digits in the original number.

3. The total obtained in Step 2 must be a number ending in zero (30, 40, 50, and soon) for the account number to be validated.

Now that we have a specification for the algorithm, we can start working on the imple-mentation.Part 2: Add a Failing Test1. Right-click on the solution node and select Add|New Project.

2. On the left pane, select Visual C#|Windows then select the Class Library item.Change the project name to Creditar.

3. In the test project, right-click on the References node and select Add References.


4. Select the Projects tab and double-click the Creditar project row to add it as areference.

5. We start by writing a first unit test for theValidatemethod, before writing or declar-ing the Validatemethod itself. Add a first unit test that verifies that the Validatemethod throws ArgumentNullExceptionwhen it receives a null reference.

[ TestMethod ][ E x p e c t e d E x c e p t i o n ( t y p e o f ( ArgumentNul lE xcep t ion ) ) ]p u b l i c vo id Nul lNumberThrowsArgumentNul lExcep t ion ( ) {

LuhnAlgor i thm . V a l i d a t e ( n u l l ) ;}

6. Right-click on the Creditar project node and select Add|Class.

7. Add a minimal implementation of the Validatemethod such that the projects willcompile.

p u b l i c s t a t i c c l a s s LuhnAlgor i thm {p u b l i c s t a t i c boo l V a l i d a t e ( s t r i n g number ) {

r e t u r n f a l s e ;}

}8. Execute the unit test and make sure that it fails.


Part 3: Run the Unit Test and Watch It Pass

1. Make a minimal change to the Validate implementation such that the unit testwill pass.


i f ( number == n u l l )throw new ArgumentNul lExcep t ion ( " number " ) ;


}2. Repeat the steps above to ensure that when a non-digit character is passed to the

Validate method, the implementation throws a ArgumentException,[ TestMethod ][ E x p e c t e d E x c e p t i o n ( t y p e o f ( ArgumentE xcep t ion ) ) ]p u b l i c vo id AThrowsArgumentException ( ) {

LuhnAlgor i thm . V a l i d a t e ( " a " ) ;}

Interestingly, the minimum change to get this test to pass is not really what onewould call a correct implementation, or even a useful implementation.



i f ( number == " a " )throw new ArgumentE xcep t ion ( " number " ) ;


}To write such an implementation really means to follow the incremental idea of thetest-driven development methodology. We could continue writing more unit teststhat will fail, in order to refine our implementation.

Part 4: Continue the iteration

1. Now that we have a passing test suite, we can refactor the code into a smarterimplementation that checks for any non-digit character



f o r e a c h ( v a r c i n number )i f ( ! Char . I s D i g i t ( c ) )

throw new ArgumentE xcep t ion ( " number " ) ;r e t u r n f a l s e ;

}}


The rest of the unit testing of the Validate method is left as an exercise. Do notforget to use the code coverage view to ensure that the unit test reach a minimum levelof basic block code coverage. This usually means that executing the unit test suite yieldsto a certain percentage of basic block coverage. In the case of this exercise, 80% is areasonable goal.

Tip: Where to get valid credit card numbers? To help you validate your implementation, youcan use this number generator [345]:

p u b l i c s t a t i c i n t [ ] CreateNumber ( i n t l e n g t h ) {Random random = new Random ( ) ;i n t [ ] d i g i t s = new i n t [ l e n g t h ] ;/ / S e t a l l b u t t h e l a s t d i g i t t o a random number ;/ / t h e l a s t d i g i t r e m a i n s z e r of o r ( i n t i = 0 ; i < l e n g t h 1 ; i ++) {

d i g i t s [ i ] = random . Next ( 1 0 ) ;}i n t sum = 0 ;boo l a l t = t r u e ;f o r ( i n t i = l e n g t h 2 ; i >= 0 ; i) {

i f ( a l t ) {i n t temp = d i g i t s [ i ] ;temp = 2 ;i f ( temp > 9) {

temp = 9 ;}sum += temp ;

}e l s e {

sum += d i g i t s [ i ] ;}a l t = ! a l t ;

}i n t modulo = sumi f ( modulo > 0) {

d i g i t s [ l e n g t h 1] = 10 modulo ;}r e t u r n d i g i t s ;

}

3 Parameterized Unit Testing

The unit test in Listing 5.2 specifies the behavior of the array list by example. Strictlyspeaking, this unit test only says that by adding a new object to an empty array list,this object becomes the first element of the list. What about other array lists and otherobjects?


1 p u b l i c void AddSpec2 (2 / / p a r a m e t e r s3 A r r a y L i s t l i s t , o b j e c t e l e m e n t ) {4 / / a s s u m p t i o n s5 PexAssume . I s T r u e ( l i s t != n u l l ) ;6 / / method sequence7 i n t l e n = l i s t . Count ;8 l i s t . Add ( e l e m e n t ) ;9 / / a s s e r t i o n s

10 P e x A s s e r t . I s T r u e ( l i s t [ l e n ] == e l e m e n t ) ;11 }

Listing 5.3. A parameterized unit test for the ArrayList class

Traditional unit tests do not take inputs. A straightforward extension is to allow pa-rameters. The result is a Parameterized Unit Test (PUT), which one can partition intofour parts: (1) parameters, (2) assumptions, (3) a method sequence and (4) assertions.

parameters represent the test input which is later passed on to other methods astheir argument values,

assumptions over the parameters can be used to shape legal test inputs, method sequence specify a scenario (as before), and assertions encode the test oracle of a unit test (as before).

Listing 5.3 is a parameterized version of the array list unit test that describes the normalbehavior of the Add method with respect to two observers, the property Count and theindexing operator []. Under the condition that a given array list is not null, this PUTasserts that after adding an element to the list, the element is indeed present at the endof the list:

This test is more general than the original test. PUTs like this one can be called withvarious input values, perhaps drawn from an attached database. Unit testing frameworksthat support PUTs sometimes refer to them as data-driven tests (for example in [2]).

Instead of stating the exemplary data values explicitly, a PUT may state assumptionsabout how valid input data must look like. Here, we assume that the list is not null.

PUTs are more general specifications than traditional unit tests: PUTs state the in-tended program behavior for entire classes of program inputs, and not just for one ex-emplary input. And yet PUTs are still easy to write since they merely state what thesystem is supposed to do, and not how to accomplish the goal.

Unlike many other forms of specification documents, PUTs are written on the levelof the actual software APIs, in the programming language of the software project. Thisallows PUTs to evolve naturally with the code against which they are written.

3.1 Separation of Concerns

Splitting the specification and test cases by parameterized unit testing is a separation ofconcerns:


1 p u b l i c c l a s s A r r a y L i s t2 . . .

3 {4 p r i v a t e O b j e c t [ ] _ i t e m s = n u l l ;5 p r i v a t e i n t _ s i z e , _ v e r s i o n ;6 . . .

7 p u b l i c v i r t u a l i n t Add ( O b j e c t v a l u e ) {8 i f ( _ s i z e == _ i t e m s . Length ) E n s u r e C a p a c i t y ( _ s i z e + 1 ) ;9 _ i t e m s [ _ s i z e ] = v a l u e ;

10 _ v e r s i o n ++;11 re tu rn _ s i z e ++;12 }13 }

Listing 5.4. ArrayList implementation in .NET

Firstly, we specify the intended external behavior of the software as PUTs; onlyhuman beings can perform this specification task.

Secondly, a tool like Pex can automatically create a test suite with high code cov-erage by determining test inputs which exercise different execution paths of theimplementation.

3.2 Coverage through Test Input Generation

Adding parameters to a unit test improves its expressiveness as a specification of in-tended behavior, but we lose concrete test cases. We can no longer execute a parame-terized test by itself. We need actual parameters. But which values must be provided toensure sufficient and comprehensive testing? Which values can be chosen at all?

Consider the code of Listing 5.4 that implements Add and the indexing operator inthe .NET base class library.

There are two cases of interest. One occurs when adding an element to an array listthat already has enough room for the new element (when the array lists capacity isgreater than the current number of elements in the array list). The other occurs whenthe internal capacity of the array list must be increased before adding the element.

We can assume that the library methods invoked by the ArrayList implementationare themselves correctly implemented (EnsureCapacity guarantees that the itemsarray is resized so its length is greater or equal size + 1), and we do not considerpossible integer overflows.

Then we only need to run two test cases to check that the assertion embedded inAddSpec2 holds for all array lists and all objects given the existing .NET implemen-tation. Two test cases are needed as there are only two execution paths through theAdd method shown above; accordingly, all inputs can be partitioned into two equiva-lence classes: one where size == items.Length holds, and one where it does nothold. Each of the two test cases below is a representative of one of the two equivalenceclasses.


[ TestMethod ]p u b l i c vo id TestAddNoOverflow ( ) {

AddSpec2 ( new A r r a y L i s t ( 1 ) , new o b j e c t ( ) ) ;}

[ TestMethod ]p u b l i c vo id TestAddWithOverf low ( ) {

AddSpec2 ( new A r r a y L i s t ( 0 ) , new o b j e c t ( ) ) ;}

No other inputs are needed to test all behaviors of Add, since any other input willexecute exactly the same paths as the two inputs mentioned above.

3.3 Theory of Parameterized Unit Tests

By adding parameters we turn a closed unit test into a universally quantified condi-tional axiom that must hold for all inputs under specified assumptions. Intuitively,the AddSpec2(. . .) method asserts that for all array lists a and all objects o, thefollowing holds:

ArrayList a, object o.(a = null) let i = a.Count in a.Add(o) , a[i] == o

where , represents sequential composition from left to right: (f , g)(x ) = g(f (x ))1.See [314] for more background information on PUTs, and [38] for an overview of thetheory and practice of algebraic specifications.

3.4 Patterns for Parameterized Unit Testing

As PUTs are really just a way to write algebraic specifications as code, many standardpatterns for algebraic specifications can be applied in the context of parameterized unittesting.

We have collected a set of such patterns in the context of Pex, which can be found inthe documentation section of the Pex website [276].

3.5 Test Driven Development by Parameterized Unit Testing

Test Driven Development [30] (TDD) is the activity of programming where all writtencode is preceeded by writing tests which specify the intended functionality of the code.

In TDD, the main purpose of writing tests is to drive the design of the API of thecode. It is simply a side effect that the result is a test suite that can serve as documen-tation and specification of the intended behavior of the code. It is straightforward toextend TDD to PUTs. In fact, it is usually more expressive to state the intended proper-ties of an API with PUTs instead of closed unit tests with exemplary data.

1 The axiom becomes more complicated when we specify side effects of sequential code pre-cisely. We explain later how to model references to objects on the heap, see Section 7.3.


When writing PUTs, the TDD process is as follows:

1. The developer writes or changes a parameterized unit test (PUT), or code that im-plements the behavior described by already existing PUTs.

2. The developer runs Pex on the PUT.3. If Pex finds errors, the developer goes back to step 1, in order to change the PUT

or fix the code, possibly with Pex Fit-It feature.4. The developer keeps the generated tests for future regression testing.

4 Selecting Test Inputs for Parameterized Unit Tests

Classic unit tests are methods without parameters, parameterized unit tests are methodswith parameters. In fact, parameterized unit tests have been around for a while now.They were already supported by Visual Studio Unit Test in Visual Studio (referred to asdata-driven tests), and by MbUnit (referred to as row tests) starting from version one,and they were recently added to JUnit [186] (referred to as theories).

Yet, it used to be the case that the user had to provide the input parameters for thosetests, as ranges, spreadsheet or database of some sort. Improper choice of inputs wouldlead to missed corner cases or a hugely redundant test suite.

With Pex, things change for the better: the user does not have to provide any inputto the parameterized unit tests. By analyzing the program behavior at runtime, Pex cangenerate inputs that matter, in the sense that those inputs will increase the coverage ofthe test suite (advanced readers can refer to Section 7 for further details).

4.1 Automated Exploratory Testing

Selecting meaningful test inputs requires a certain understanding of the code under test,which in turn requires an understanding of what the relevant parts of the (potentiallyhuge) code under test are.

Exploratory Testing [189,17] (ET) is an incremental process during which the testerlearns more and more about the actual behavior of the code. Another characterization ofET is that it is test design and test execution at the same time. Together with experienceand creativity the tester can craft more and better tests.

Pex uses dynamic symbolic execution [141], a technique that works in a way similarto ET. The tool executes the code multiple times and learns about the program behaviorby monitoring the control and data flow. After each run, it picks a branch that was notcovered previously, builds a constraint system (a predicate over the test inputs) to reachthat branch, then uses a constraint solver to determine new test inputs, if any. The testis executed again with the new inputs, and this process repeats. On each run, Pex mightdiscover new code and dig deeper into the implementation. In this way, Pex explores thebehavior of the code. We also refer to this process as Automated Exploratory Testing.

For the advanced reader, Section 7 discusses dynamic symbolic execution.Exploratory testing example. We will apply exploratory testing to test a simple method(see Listing 5.5) that takes two integers as an input and prints different strings to theconsole based on those values. We will manually go through all the steps involved inthe analysis. (Pex would perform a similar analysis, only fully automatically.)


1 void Bar ( i n t i , i n t j ) {2 i f ( i < 0) {3 Conso l e . W r i t e L i n e ( " l i n e 3 " ) ;4 i f ( j == 123)5 Conso l e . W r i t e L i n e ( " l i n e 5 " ) ;6 }7 e l s e8 Conso l e . W r i t e L i n e ( " l i n e 8 " ) ;9 }

Listing 5.5. A method to explore

One way to explore the possible behaviors of this method is to throw different valuesat Bar and analyze the output to see what is happening.

Iteration 1: pick arbitrary value. Let us create a unit test that does exactly that and stepinto the debugger. Since we do not really know anything about the Bar method yet, wesimply pick 0 for i and j .

[ TestMethod ]vo id Zero ( ) {

Foo . Bar ( 0 , 0 ) ;}

When we reach the statement Console.WriteLine("line 8"); on line 8, we canfigure out that we took this branch because the condition i < 0 on line 2 evaluated tofalse. With this we let the execution continue (and the test finishes successfully).

line 2, i 0, uncovered branch

Iteration 2: flip the last condition In the previous run, we have remembered that somecode was not covered on line 3. We also know that this code path was not coveredbecause the condition i < 0 evaluated to false. At this point, we usually intuitivelyfigure out a value of i in our head, to make this condition true. In this case, we needto solve find i such that i < 0. Let us pick 1.

[ TestMethod ]vo id MinusOne ( ) {

Foo . Bar (1 , 0 ) ;}

We run the test under the debugger. As expected on line 2, the condition evaluates totrue, and the program takes the other branch (compared to the one taken in the previoustest). The program continues and reaches line 4 where another if-statement is evaluated.The condition j = 123 evaluates to false, so we remember the following:

line 4, j = 123, uncovered branchThe program continues to run and finishes.


Iteration 3: path condition + flipped condition. There are still some uncovered branchesto cover in the method, guarded by the condition at line 4. To be able to cover this code,we need two things:

1. reach line 4: i < 02. make the condition in line 4 evaluate to true: j = 123

So to cover the last statement in the method, we need to find parameter values such thati < 0 j = 123.

Let us pick i = 1 and j = 123.[ TestMethod ]vo id MinusOneAndOneTwoThree ( ) {

Foo . Bar (1 , 1 2 3 ) ;}

The test executes and prints line 5 as we wanted. At this point, we have fullycovered the behavior of Bar.

4.2 Exercises

Exercise 3. Getting started with Pex in Visual Studio

Part 1: Adding Pex to a Project

1. Add a reference to the Microsoft.Pex.Framework.dll assembly to the testproject. In the Add Reference dialog, select the .NET pane, then scroll down toMicrosoft.Pex.Framework,

Part 2: Creating a Parameterized Unit Test

1. In the HelloWorldTest, add a new public instance method Parameterized-Test that takes an int parameter. Mark this method with the PexMethodAttri-bute, written as [PexMethod] in C#.

[ PexMethod ]p u b l i c vo id P a r a m e t e r i z e d T e s t ( i n t i ) {

i f ( i == 123)throw new ArgumentE xcep t ion ( " i " ) ;

}


Tip: Adding Using Clauses Automatically Visual Studio can automatically add the missingusing clause: Move the editing caret on the [PexMethod] attribute. You will see a little redrectangle, located at the bottom right of PexMethod. Click on the little red rectangle. You geta context menu with two entries: The first will add a using clause to your source code, and thesecond will embed the fully qualified namespace to this occurrence of PexMethod. Just pressEnter to insert the using clause. The keyboard shortcut to open this context menu is Ctrl +..

Tip: Using Snippets Pex also provides snippets to create a new parameterized unit test skeleton.In the editor, write pexm and then press the tab-key. This will expand the snippet.

Part 3: Run the Parameterized Unit Test1. Move the mouse cursor inside the ParameterizedTest method, right-click and

select the Run Pex Exploration menu item.

2. Pex automatically displays the Pex Results window. Most of your interactions withPex will be through this window.

3. Each row in the table corresponds to a generated test for the current exploration.Each row contains

an icon describing the status of the test (passing, failing), a number indicating how often Pex had to execute the parameterized unit test

with different input values in order to arrive at this test, a summary of the exception that occurred, if any


Pex also automatically logs the values of the input parameters of the test. Notethat often Pex runs the parameterized unit test several times until Pex outputs thenext test. The rationale behind this behavior is that Pex explores different execu-tion paths of the program, but it only outputs a new test when this test increasesthe coverage (arc coverage, to be precise; see Section 7.9 for more details). Manyexecution paths may result in the same coverage.

4. When exploring the ParameterizedTest that we wrote earlier, Pex generatestwo unit tests. Each unit test can be accessed by selecting the corresponding rowand clicking on the Go to generated test link.

[ TestMethod ][ PexGeneratedBy ( t y p e o f ( T e s t C l a s s ) ) ]p u b l i c vo id P a r a m e t e r i z e d T e s t 0 1 ( ) {

t h i s . P a r a m e t e r i z e d T e s t ( 0 ) ;}[ TestMethod ][ PexGeneratedBy ( t y p e o f ( T e s t C l a s s ) ) ][ P e x R a i s e d E x c e p t i o n ( t y p e o f ( ArgumentE xcep t ion ) ) ]p u b l i c vo id P a r a m e t e r i z e d T e s t 0 2 ( ) {

t h i s . P a r a m e t e r i z e d T e s t ( 1 2 3 ) ;}

The TestMethod attribute indicates that the generated methods are unit tests, thePexGeneratedBy attribute indicates that the test was generated by Pex by exploringparameterized unit tests in a particular test class, and the PexRaisedExceptionindicates that this test raised an (unexpected) exception.

Exercise 4. Using Pex from the Command Line. In Section 4.2 we learned how to usePex in the Visual Studio environment. In this exercise, we will learn how to drive Pexfrom the command line and how to read the HTML reports generated by Pex.

Before running Pex, you should have a .NET assembly (a .NET executable, endingin .dll or .exe) that contains a class annotated with the PexClassAttribute , contain-ing a public instance method annotated with the PexMethodAttribute. We will use theListing 5.6, which corresponds to the example that we created with Visual Studio inExercise 2.5. You can build it with the standalone C#-compiler csc.exe, Visual C#2005/2008 Express Edition, or any other C# development environment. In any case,you need to reference the Microsoft.Pex.Framework assembly.

Tip: Other test frameworks Pex works best with the unit test framework of Visual Studio,but Pex can also generate tests for other unit test frameworks. Pex detects the intended unit testframework by inspecting the referenced assemblies in your test project.

We run Pex from the command line as follows.

1. Open a command prompt window.2. Go to the build folder of the test project,


1 u s i n g System ;2 u s i n g M i c r o s o f t . Pex . Framework ;3

4 [ P exClass ]5 p u b l i c p a r t i a l c l a s s T e s t C l a s s {6 [ PexMethod ]7 p u b l i c void P a r a m e t e r i z e d T e s t ( i n t i ) {8 i f ( i == 123)9 throw new ArgumentE xcep t ion ( " i " ) ;

10 }11 }

Listing 5.6. Self-contained parameterized unit test code

Tip: Getting the path of a document in Visual Studio One can get the full path of an opendocument in Visual Studio by right-clicking on the tab and select Copy Full Path.

3. Apply the Pex command line program, pex.exe, on the test project assembly.This will run all the parameterized unit tests in that assembly (you may need tochange name of the .dll file as appropriate):

pex bin\Debug\TestProject1.dllPex runs and logs its activity to the console. At the end of the run, it automaticallyopens a detailed HTML report.

Part 1: Command Line Output The console output contains different aspects. The pre-cise output may vary, depending on your version of Pex and .NET.

1. Pex inspects the test project and launches a second process. (In this process, Pexacts as a profiler in order to instrument code so that Pex can monitor a taken exe-cution path precisely.)

1 Microsoft Pex v0.18 -- http://research.microsoft.com/pex2 Copyright (c) Microsoft Corporation 2007-2009.3 All rights reserved.4

5 instrumenting...2. At this point, the instrumented process is running. Pex is loading the parameterized

tests. Any issue in the test metadata will be logged during this phase.


6 launched Pex x86 Edition on .NET v2.0.507277 [reports] report path:8 reports\TestProject1.71115.170912.pex9 00:00:00.0> starting execution

10 00:00:00.1> reflecting tests3. In the next phase, Pex explores all parameterized unit tests.

11 00:00:02.8> TestProject112 00:00:02.8> TestProject1.TestClass13 00:00:12.3> TestClass.ParameterizedTest(Int32)

4. At each new test, Pex writes a line that describes the test:the test ParameterizedTest02 was generated on the second run and raised aArgumentException.

14 [test] (run 1) ParameterizedTest00 (new)15 [coverage] coverage increased from 0 to 2 blocks16 [test] (run 2) ParameterizedTest01 (new),17 Exception: Exception of type System.Exception was thrown.18 [coverage] coverage increased from 2 to 4 blocks

Tip: How does Pex name generated tests? The generated test names are created by manglingthe exploration method name, plus a number.

5. At the end of the exploration, Pex also gives a basic block coverage summary.19 [dynamic coverage] 4/4 block (100.00 percent)

6. Once all parameterized unit tests have been processed, Pex displays some statistics,then renders the HTML report. In this run, seven tests were generated, one of whichwas considered as a failing test:

20 00:00:13.2> [finished]21 -- 2 generated tests, 1 failing, 2 new

No errors or warnings were logged by the Pex infrastructure:22 -- 0 critical errors, 0 errors, 0 warnings

Every time you run Pex, it generates a new folder in the reports directory tohold all the generated files (and by default, the folders get recycled after a while;in other words, the generated reports have only a limited lifetime, and you shouldcopy a report that you want to preserve). The generated folder name is constructedfrom the test assembly name and a time stamp.

Tip: Where are the generated tests? Pex creates a subdirectory, here it would bereports\TestProject1.71115.170912.pex\tests, which contains the generatedtests as code.

23 [reports] generating reports...24 [reports] html report:25 reports\TestProject1.71115.170912.pex\pex.html

7. Finally, Pex displays the overall verdict, success or failure.26 EXPLORATION SUCCESS


Part 2: Command Line Filters The command line supports a rich set of filters to selectwhich exploration to run: by namespace, by type name, by method name, by category,and so on.

by namespace,pex /nf:TestProject ...

by type name,pex /tf:HelloWorld ...

by method name,pex /mf:Add ...

by test suite,pex /sf:checkin ...

the explorations that had errors in the last runs,pex /lef ...

of course, you can combine all those filters together,pex /nf:TestProject /tf:HelloWorld /mf:Add ...

By default, a partial case-insensitive string match is used. To have a precise match,add a bang (!) at the end of the filter:

pex /tf:HelloWorldTest!

Part 3: Using Pex HTML report The HTML reports gives a tabular view of each as-sembly/fixture/exploration. Overall statistics, including the number of fixtures and gen-erated tests, are highlighted in the title,

1. Each exploration has a little menu to display associated data.


Tip: Do I have access to the HTML reports in Visual Studio? To enable HTML report gener-ation, go to Tools|Options then locate the Pex - General category, click on the Reports option.

You need to run Pex again to actually generate a report. Afterwards, to open the HTML report,click on the Open Report link in the yellow bar.


2. The parameter values displays a table where each row contains the values of theparameters for a particular test. This table is very similar to the Pex Results viewin Visual Studio.

3. The log item opens a detailed activity log of the exploration process. The log is avery important tool to understand in detail which events Pex encountered, and youwill get familiar with it soon.

4. The log of this test starts with a repro command that can be used to execute thatparticular exploration from the command line. Then, the generated tests are logged.

5. The coverage link opens the code coverage report document.


6. You can browse through the coverage data by assemblies, fixture and methods.

7. You can also view the coverage by source files.

8. Each source file is colored according to the legend found at the top of the page.user code under test (covered) Covered statements of the code-under-test.user code under test (uncovered) Statements of the code-under-test that were not

covered.user code or test (covered) Statements that were covered, but that were not part

of the code-under-test.user code or test (uncovered) Statements that were neither covered nor under test.tagged Statements which were covered, and which were tagged with some inter-

esting information during the exploration process. Hover with the mouse overa tagged statement to see more information.

We will later discuss how to configure which parts of the code are under test.

9. In this particular case, the ParameterizedTest method was fully covered,

Exercise 5. Parameterized Unit testing for the Luhn Algorithm We revisit the Luhnalgorithm that we tested in Section 2.5 and write parameterized unit tests for it.


Part 1: Setting Up the Code Instrumentation

1. Add a parameterized unit test that specifies that a valid credit card number, accord-ing to the Luhn algorithm, does not contain any non-digit characters.

[ PexMethod ]p u b l i c vo id ContainsNumbers ( s t r i n g number ) {

PexAssume . I s T r u e ( LuhnAlgor i thm . V a l i d a t e ( number ) ) ;

P e x A s s e r t . T r u e F o r A l l ( number , d e l e g a t e ( c h a r c ) {r e t u r n c h a r . I s D i g i t ( c ) ;

} ) ;}

2. Run the test and analyze the results. As you may see, Pex only generated a single(trivial) test with a null reference as the number string.

3. The problem is that Pex did not analyze the Validate method. (Pex can onlygenerate relevant test inputs for those portions of the code which get instrumentedand can be monitored, and since the instrumentation is quite expensive, Pex doesnot instrument all code by default.) You need to tell Pex which assemblies or typesneed to be instrumented. This is a process that is usually done once.

In our case, the product assembly is not instrumented, thus Pex could not buildthe constraints to explore the behavior of the Validate method, which eventuallyleads to poor code coverage of the generated test suite.

4. Pex shows important issues such as calls to methods that were not instrumented ina red bar.

5. Click on the link Uninstrumented Methods to see the list of relevant methodsthat were not instrumented. When you click on a method in the list, Pex offersseveral actions that can be taken to get rid of the warning. Since we want to test theValidate method, you want to click on the Instrument assembly link.

6. This action will add a custom attribute in the test project that tells Pex that theproduct assembly should be instrumented:


[ as sembly : P exIns t rumen tAs se mb ly ( " C r e d i t a r " ) ]

Pex adds a special file PexAssemblyInfo.cs to hold the assembly level at-tributes. (Pex persists all project-specific settings as attributes.)

7. Run Pex again and analyze the results. The coverage should be much better.

Part 2: Binding the Tests and the Code Under Test You should tell Pex what is the codeunder test, so that Pex can focus its analysis on that code, and show a relevant dynamiccoverage graph. Pex provides several ways to specify this information.

1. The PexClassAttribute has a special constructor that takes a type. This constructorcan be used to specify the type under test of a particular test fixture. This informa-tion is used by Pex to prioritize the exploration. Update the LuhnAlgorithmTestclass to specify that it is testing the LuhnAlgorithm class.

[ T e s t C l a s s , P exClass ( t y p e o f ( LuhnAlgor i thm ) ) ]p u b l i c p a r t i a l c l a s s L uhnAlgor i t hmT es t {

2. Because we know that the project Creditar.Tests is the test project for Credi-tar, we can add a custom attribute that will provide this information to Pex.

[ as sembly : PexAssemblyUnderTest ( " C r e d i t a r " ) ]

It is an assembly-level attribute that must be placed before the namespace decla-ration in C#.

Part 3: Using the Wizard to Get Started Pex provides a code generation wizard thatcan automatically produce parameterized test stubs for the entire public API of a class.These stubs can be used as a starting point to write more elaborate scenarios.

1. Right-click inside the LuhnAlgorithm class and select Pex|Create ParameterizedUnit Test Stubs

2. The Pex wizard will compile and analyze the LuhnAlgorithm class to produce thetest stubs. The generated files will automatically be added to the test project usingthe information provided by the PexAssemblyUnderTestAttribute attribute.


Exercise 6. Instrumentation Configuration Pex can only generate a test suite with highcode coverage if Pex monitors the relevant parts of the code. Therefore, it is most im-portant to configure correctly which types Pex should instrument.

Consider the following parameterized unit test.

[ PexMethod ( MaxBranches = 2 0 0 0 ) ]p u b l i c vo id T e s t ( s t r i n g s ) {

DateTime d t = DateTime . P a r s e ( s ) ;PexObserve . ValueForViewing ( " d t " , d t ) ;

}The MaxBranches setting makes sure that Pex does not stop too early. We will explainthese bounds later, see Section 7.10.

When Pex generates tests, it will only generate a single test at first. However, we doget a warning that some methods were not instrumented.

When you click on Uninstrumented Methods, you see in the log view the list ofuninstrumented methods.

You can select one of them, and click on the link Instrument type to tell Pex that itshould instrument this type in the future. (If you know that the code is irrelevant to thecode that we want to cover, you can also click on Ignore uninstrumented method.)Pex will insert custom attributes such as the following for you.

u s i n g M i c r o s o f t . Pex . Framework . I n s t r u m e n t a t i o n ;[ a s sembly : P e x I n s t r u m e n t T y p e ( " m s c o r l i b " ,

" System . Da teT imeP ar se " ) ][ a s sembly : P e x I n s t r u m e n t T y p e ( " m s c o r l i b " ,

" System . __DTStr ing " ) ]After you instruct Pex to instrument a type, you have to re-run Pex to see the effects

of more instrumentation. In turn, you might get more uninstrumented method warnings.Can you determine the set of types that Pex must instrument in order to generate

a valid DateTime string? It is usually easier when you want to test your own codewhose structure you know. Hint: Work in a loop, where you add one type to the list ofinstrumented type which has the word DateTime in the name, and then re-run Pex tosee what changed.


Tip: Coverage filtering The type DateTime is defined in mscorlib. Pex normally does notreport coverage data achieved in this basic library. We can tell Pex to include coverage data withthe following assembly-level attribute:

u s i n g M i c r o s o f t . Pex . Framework . Coverage ;u s i n g M i c r o s o f t . E x t e n d e d R e f l e c t i o n . Coverage ;[ a s sembly : P e x C o v e r a g e F i l t e r A s s e m b l y (

PexCoverageDomain . UserCodeUnderTest ," m s c o r l i b " ) ]

Exercise 7. Regular Expressions (optional): Try out your favorite regular expressionand see if Pex can generate an input string that matches that expression:

[ PexMethod ]p u b l i c vo id MatchRegex ( [ PexAssumeNotNull ] s t r i n g v a l u e ) {

i f ( Regex . I sMatch ( va lue , @"^[09afAF ] + [ \ r \ n ] $ " ) )Conso l e . W r i t e L i n e ( " found i t ! " ) ;

}The [PexAssumeNotNull] attribute indicates that Pex should pass null for the

value parameter. This is a simple case of an assumption on test inputs. See Section 7.7for more details.

Exercise 8. Binary Encoding (optional): An encoding is a reader-writer between bytearrays and char arrays.

A simple example of encoding simply copies the bits of a char to two byte, and viceversa.

The .NET Encoding class is defined in the System.Text namespace and has fourabstract methods that need to be overloaded.

Using the following PUT, write an implementation for GetChars and GetChar-Count

[ PexMethod ]p u b l i c vo id CharCountMatching (

[ PexAssumeNotNull ] Encoding encoding ,b y t e [ ] by t es ,i n t index ,i n t c o u n t ) {i n t charCount = encod ing . GetCharCount ( by t es , index , c o u n t ) ;c h a r [ ] c h a r s = encod ing . GetChars ( by t es , index , c o u n t ) ;

A s s e r t . AreEqual ( charCount , c h a r s . Length ) ;}

You can start from the Listing 5.7 which contains a sloppy implementation of thosemethods.

Exercise 9. Compression and Decompression (optional): Write a run-length encoding(RLE) compression and decompression algorithm, that take arrays of bytes as input andoutput. Listing 5.8 describes the signature of the RLE implementation.


1 p u b l i c c l a s s S loppyBinaryE nco din g : Encoding {2 c o n s t i n t BytesPe rChar = 2 ;3

4 p u b l i c o v e r r i d e i n t GetCharCount (5 byte [ ] by t es , i n t index , i n t c o u n t ) {6 re tu rn c o u n t / 2 ;7 }8 p u b l i c o v e r r i d e i n t GetChars (9 byte [ ] by t es , i n t b y t e I n d e x , i n t byteCount ,

10 char [ ] c h a r s , i n t c h a r I n d e x ) {11 i n t j = c h a r I n d e x ;12 f o r ( i n t i = 0 ; i < by teCount ; i += 2) {13 c h a r s [ j ++] =14 ( char ) ( b y t e s [ b y t e I n d e x + i ]


Exercise 10. String Interview Question (optional) A classic theme in programming jobinterviews is on manipulating strings. Here is one problem that we could test with Pex:

Given an array of characters, with spaces, replace spaces with %20. You have extraspace on the end of the array. (No additional memory and do it as close to O(n) aspossible).

Write an implementation of EscapeSpaces. Write a PUT and use Pex to validate your implementation.

5 Dealing with the Environment

Each unit test, whether parameterized or not, should test a single feature, so that a failingunit test identifies the broken feature as concisely as possible. Also, the fewer systemcomponents a test interacts with, the faster it will run.

However, in practice it is often difficult to test features in isolation: The code maytake a file name as its input, and use the operating system to read in the contents of thefile. Or the test may need to connect to another machine to fulfill its purpose.

The first step towards making the code testable is to introduce abstraction layers.For example, the following Parse method is not testable in isolation, since it insists oninteracting with the file system.

p u b l i c vo id P a r s e ( s t r i n g f i l eName ) {S t reamReader r e a d e r = F i l e . OpenText ( f i l eName ) ;s t r i n g l i n e ;w h i l e ( ( l i n e = r e a d e r . ReadLine ( ) ) != n u l l ) {

. . .

}}

The parser in the following code is better testable, since the actual parsing logic canbe driven from any implementation of the abstract StreamReader class. In particular,it is no longer necessary to go through the file system to test the main code.

p u b l i c vo id P a r s e ( s t r i n g f i l eName ) {t h i s . P a r s e ( F i l e . OpenText ( f i l eName ) ) ;

}

p u b l i c vo id P a r s e ( S t r eamReader r e a d e r ) {s t r i n g l i n e ;w h i l e ( ( l i n e = r e a d e r . ReadLine ( ) ) != n u l l ) {

. . .

}}

Abstraction from the environment is necessary if you want to have a Pex-testabledesign.


5.1 Mocks, Stubs

When the code is written with abstraction layers, mock objects [226] can be used tosubstitute parts of the program that are irrelevant for a tested feature. Mock objectsanswer queries with fixed values similar to those that the substituted program wouldhave computed.

Today, developers usually define the behavior of mock objects by hand. By behavior,we mean the return values of mocked methods, what exceptions they should throw, andso on. Several frameworks [343] exist which provide stubs. Stubs are trivial implemen-tations of all methods of an interface or a class. Stubs do not perform any actions bythemselves and usually just return some default value. The behavior of the stubs muststill be programmed by the developer. (A capture-replay approach is used in [292] todistill actual behavior of an existing program into mock objects.)

5.2 Stubs Framework in Pex

In fact, Pex comes with its own Stubs framework to make writing stubs easier. Thisframework also allows to detour legacy code that communicates with the environmentand does not provide an encapsulated abstraction layer. In the remainder of this tutorial,we will not use any framework for mocks or stubs, not even the one that comes with Pex,but we will define everything required by hand to illustrate the general mechanisms.Tutorials and more documentation on how to leverage the stubs framework that comeswith Pexcan be found on the Stubs website [277].

5.3 Parameterized Mock ObjectsWhen manually writing mock objects, some of the main questions are: What valuesshould the mock object return? How many versions of a mock object do I need to writeto test my code thoroughly?

We have seen earlier how parameterized unit tests are a way to write general teststhat do not have to state particular test inputs. In a similar way, parameterized mockobjects are a way to write mock objects which do not have just one particular, fixedbehavior.

Consider the method AppendFormat of the StringBuilder class in the .NETbase class library. Given a string with formatting instructions, and a list of values to beformatted, it computes a formatted string. For example, formatting the string "Hello{0}!" with the single argument "World" yields the string "Hello World!".

p u b l i c S t r i n g B u i l d e r AppendFormat (I F o r m a t P r o v i d e r p r o v i d e r ,s t r i n g forma t , o b j e c t [ ] a r g s ) {

. . .

}The first parameter of type IFormatProvider provides a mechanism for retrieving

an object to control formatting according to the MSDN documentation:


p u b l i c i n t e r f a c e I F o r m a t P r o v i d e r {o b j e c t GetFormat ( Type fmtType ) ;

}A non-trivial test calling AppendFormat needs an object that implements IFor-

matProvider. While the Stubs framework [277] that comes with Pex can generateimplementations for interfaces automatically, we will illustrate in the following how towrite such an implementation by hand, but leaving to Pex how it should behave.

p u b l i c c l a s s MockFormatProv ide r : I F o r m a t P r o v i d e r {p u b l i c o b j e c t GetFormat ( Type fmtType ) {

v a r c a l l = PexChoose . FromCal l ( t h i s ) ;r e t u r n c a l l . ChooseResu l t < o b j e c t > ( ) ;

}}

The mock method GetFormat obtains from the global PexChoose a handle calledcall that represents the current method call. The PexChoose provides the values whichdefine the behavior of the mocked methods, for example their return values.

When the test case is executed, ChooseResult will initially return some simplevalue, for example null for reference types. Pex symbolic analysis will track how thevalue obtained from ChooseResult is used by the program, just as Pextracks all othertest inputs.

Depending on the conditions that are checked on the value obtained from Choose-Result, Pex will execute the test case multiple times, trying other values that will bedifferent from null.

You may change the code of the mock type to allow more diverse behavior, forexample adding the choice to throw an exception, perform a callback, or change someaccessible state. For example, you can insert the following lines in the GetFormatmethod.

v a r c a l l = PexChoose . FromCal l ( t h i s ) ;i f ( c a l l . ChooseThrowExcept ion ( ) )

throw c a l l . ChooseE xcep t ion ( ) ;r e t u r n c a l l . ChooseResu l t < o b j e c t > ( ) ;

The choice whether to throw an exception will cause the exploration to consider twoexecution paths. Furthermore, if the caller of GetFormat would distinguish differentexception types, for example by having several catch statements, Pex may exploreeven more execution paths.

As mentioned before, Pex will track the usage of the values obtained from Choose-Result, and may execute the program with several different values to this end. The fol-lowing call to GetFormat occurs in AppendFormat after checking provider!=null:

c f = ( I C u s t o m F o r m a t t e r ) p r o v i d e r. GetFormat ( t y p e o f ( I C u s t o m F o r m a t t e r ) ) ;

Depending on the result of GetFormat, the cast to ICustomFormatter might fail.Pex understands this type constraint, and Pex generates a test case with the followingmock object behavior:


MockFormatProvide r m = new MockFormatProv ider ( ) ;PexChoose . NewTest ( )

. OnCal l ( 0 , " MockFormatProv ider . GetFormat ( Type ) " )

. R e t u r n s (m) ;

Here, Pex creates a mock object and instructs the oracle that during the execution ofa unit test the first call to m.GetFormat should return the mock object itself! (Thetest cases that Pex generate are always minimal, this is an example of how Pex triesto use as few objects as possible to trigger a particular execution path.) This particularmock object behavior will cause the cast to fail, since MockFormatProvider does notimplement ICustomFormatter.

5.4 Parameterized Mock Objects with Assumptions

Unconstrained mock objects can cause the code to behave in unexpected ways. Justas you can state assumptions on the arguments of parameterized unit tests, you canstate assumptions on the results of mock object calls. For example, the author of theIFormatProvider interface probably had the following contract in mind:

p u b l i c o b j e c t GetFormat ( Type fmtType ) {v a r c a l l = PexChoose . FromCal l ( t h i s ) ;o b j e c t r e s u l t = c a l l . ChooseResu l t < o b j e c t > ( ) ;

/ / c o n s t r a i n i n g r e s u l tPexAssume . I s T r u e ( r e s u l t != n u l l ) ;PexAssume . I s T r u e ( fmtType .

I s A s s i g n a b l e F r o m ( r e s u l t . GetType ( ) ) ) ;r e t u r n r e s u l t ;

}

5.5 Dependency Injection

Dependency Injection [122], also known as Inversion Of Control is a design pattern thathelps building components that are mockable.

Let us illustrate this concept with the implementation of a client application thatvalidates a credit card number.

Credit card validation client. The client is a simple graphical user interface that lets theuser input a credit card number of query for its validity. Then, the client should querythe web service to validate the credit card number and display the result to the user.


A first attempt. The dialog window usually has a Show window to pop up the dialog,a Number property to access to the number string value and a Status property toupdate its display:

p u b l i c c l a s s C r e d i t a r D i a l o g {

p u b l i c boo l Show ( ). . .

p u b l i c s t r i n g Number. . .

p u b l i c s t r i n g S t a t u s. . .

}

A first attempt at writing the validator would be the following:p u b l i c c l a s s C r e d i t a r C l i e n t {

p u b l i c vo id E xecu t e ( ) {C r e d i t a r D i a l o g d i a l o g = new C r e d i t a r D i a l o g ( ) ;i f ( d i a l o g . Show ( ) ) {

boo l v a l i d = LuhnAlgor i thm . V a l i d a t e (d i a l o g . Number ) ;

i f ( v a l i d )d i a l o g . S t a t u s = " v a l i d a t e d " ;

e l s ed i a l o g . S t a t u s = " i n v a l i d a t e d " ;

}}

}The implementation of the Executemethod causes several issues when it comes totesting it. The control flow depends on the Show method of the CreditarDialogdialog window:

C r e d i t a r D i a l o g d i a l o g = new C r e d i t a r D i a l o g ( ) ;i f ( d i a l o g . Show ( ) ) {

This means that the test will pop up a dialog window and we will need complexautomated tools to artificially click on the buttons of the dialog. In that sense, wewill spend a large amount of effort to test a functionality that is not directly relatedto the CreditarClient method.

A similar problem arises with the validate of the credit card number where wedirectly call the Validate method of the LuhnAlgorithm type:

bool v a l i d = LuhnAlgor i thm . V a l i d a t e (d i a l o g . Number ) ;

To work around these problems, we need to add an abstract layer between theCreditarClient class and its dependencies, the user interface and the validatorservice.


Extracting dependencies. We do this by extracting an interface for each dependency(we already have an interface for the credit card validation service):

p u b l i c i n t e r f a c e I C r e d i t a r D i a l o g {bool Show ( ) ;s t r i n g Number { g e t ; }s t r i n g S t a t u s { g e t ; s e t ; }

}Each dependency is then injected into the CreditarClient:

p u b l i c p a r t i a l c l a s s C r e d i t a r C l i e n t {I C r e d i t a r D i a l o g d i a l o g ;I C r e d i t C a r d V a l i d a t o r v a l i d a t o r ;

p u b l i c vo id E xecu t e ( ) {i f ( t h i s . d i a l o g . Show ( ) ) {

boo l v a l i d =t h i s . v a l i d a t o r . V a l i d a t e ( d i a l o g . Number ) ;

i f ( v a l i d )t h i s . d i a l o g . S t a t u s = " v a l i d a t e d " ;

e l s et h i s . d i a l o g . S t a t u s = " i n v a l i d a t e d " ;

}}

}

Injecting dependencies. The last problem that remains is that we need a way to set thedialog and validator fields in the CreditarClient class. There are many waysto implement this feature, the following are common patterns:Constructor Injection. Each dependency is passed in the class constructor, thus the

inversion of control name of the pattern,p u b l i c c l a s s C r e d i t a r C l i e n t {

I C r e d i t a r D i a l o g d i a l o g ;I C r e d i t C a r d V a l i d a t o r v a l i d a t o r ;

p u b l i c C r e d i t a r C l i e n t (I C r e d i t a r D i a l o g d i a l o g ,I C r e d i t C a r d V a l i d a t o r v a l i d a t o r ) {t h i s . d i a l o g = d i a l o g ;t h i s . v a l i d a t o r = v a l i d a t o r ;

}

. . .

}Service Locator. The class instance is hosted in a container that can be queried through

a message for particular services. In .NET, this pattern is implemented in the Sys-tem.ComponentModel namespace. A Component element can query for a serviceusing the GetService method:


1 i n t Compl i ca t ed ( i n t x , i n t y ) {2 i f ( x == O b f u s c a t e ( y ) )3 throw new R a r e E x c e p t i o n ( ) ;4 re tu rn 0 ;5 }6

7 i n t O b f u s c a t e ( i n t y ) {8 re tu rn (100 + y ) 5679 }

Listing 5.9. Code that is difficult to analyze

p u b l i c c l a s s C r e d i t a r C l i e n t : Component {p u b l i c vo id E xecu t e ( ) {

I C r e d i t a r D i a l o g d i a l o g =( I C r e d i t a r D i a l o g )t h i s . G e t S e r v i c e ( t y p e o f ( I C r e d i t a r D i a l o g ) ) ;

I C r e d i t C a r d V a l i d a t o r v a l i d a t o r =( I C r e d i t C a r d V a l i d a t o r )t h i s . G e t S e r v i c e ( t y p e o f ( I C r e d i t C a r d V a l i d a t o r ) ) ;

. . .

}}

6 Background on White Box Software Testing (Advanced)In general, all program analysis techniques fall somewhere in between the followingtwo ends of the spectrum:

Static analysis techniques verify that a property holds on all execution paths. Sincethe goal is program verification, these techniques are usually overly conservativeand flag possible violations as errors (false positives).

Dynamic analysis techniques verify that a property holds on some execution paths.Testing is a dynamic technique that aims at detecting bugs, but cannot usually proofthe absence of errors. Thus, these techniques often fail to detect all errors.

It is often not possible to detect bugs precisely when applying only static analysis, oremploying a technique that is not aware of the code structure. Consider Listing 5.9.

Static analysis techniques tend to be overly conservative, and the non-linear integerarithmetic present in Obfuscate causes most static analysis techniques to give upand issue a warning about a potential error in Complicated.

Random testing techniques have very little chance of finding a pair of x and y valuesthat triggers the exception.

Pex implements an analysis technique which falls in between these two extremes: dy-namic symbolic execution, a white box test generation technique. Similar to static anal-ysis techniques, Pex proves that for most feasible paths (those within specified explo-ration bounds) a property holds. Similar to dynamic analysis techniques, Pex only re-ports real errors, and no false positives.


6.1 Specification Techniques

All program analysis techniques try to proof and/or falsify certain specified propertiesof a given program. There are different ways to specify program properties:

API Contracts (including Spec# [23] and Eiffel[248]) specify the behavior of indi-vidual API actions from the implementations point of view. Their goal is to guar-antee robustness, in the sense that operations do not crash and data invariants arepreserved. A common problem of API contracts is their narrow view on individ-ual API actions, which make it hard to describe system-wide protocols. Complexextensions like model-fields are necessary to raise the abstraction level.

Unit Tests (JUnit [186], NUnit [322], and so on) are exemplary usage scenariosfrom the point of view of a client of the API. Their goal is to guarantee functionalcorrectness, in the sense that the interplay of several operations behaves as intended.A common problem of unit tests is that they are detached from the details of theimplementation; measuring the code coverage achieved by the unit tests gives onlya rough idea of how adequate the unit tests are.

Pex enables Parameterized Unit Testing, which unites both. Supported by a test-inputgeneration tool like Pex, this methodology combines the client and the implementa-tion point of view. The functional correctness properties (parameterized unit tests) arechecked on most cornercases of the implementation (test input generation).

6.2 The Testing Problem

Starting from parameterized unit tests as specification, we can state the testing problemas follows.

Given a sequential program P with statements S , compute a set of programinputs I such that for all reachable statements s in S there exists an input i inI such that P(i) executes s.

Remarks:

By sequential we mean that the program is single-threaded. We consider failing an assertion, or violating an implicit contract of the execution

engine (for example raising NullReferenceException when null is derefer-enced) as special statements.

Since reachability is not decidable in general, we aim for a good approximationin practice: high coverage of the statements of the program. Instead of statementcoverage, other coverage metrics like arc coverage can be used.

6.3 White-box Test Input Generation

White box testing means leveraging information about how a software system is imple-mented in order to validate or falsify certain properties. White box testing may involvethe analysis of data flow, control flow, exception handling, or other details of the im-plementation. In order to obtain the necessary information, white box testing requires


access to the softwares source code, or the compiled binary. Pex is a white-box analysistool that analyzes the compiled instructions sequence of the program (the .NET MSILinstructions).

The opposite of white box testing is black box testing, which usually amounts tousing randomly generated test data.

6.4 Symbolic ExecutionPex implements a white box test input generation technique that is based on the con-cept of symbolic execution. Pex goal is to automatically and systematically produce theminimal set of actual parameters needed to execute a finite number of finite paths. Ingeneral, symbolic execution works as follows: For each formal parameter, a symbolicvariable is introduced. When a program variable is updated to a new value during pro-gram execution, then this new value is often expressed as an expression over symbolicvariables. When a statement in the program has more than one possible successor, ex-ecution is forked into two paths. For each code path explored by symbolic execution,a path condition is built over symbolic variables. For example, the Add-method of theArrayList implementation shown in Listing 5.4 contains an if-statement whose con-dition is this. items.Length == this. size (where the field items denotesthe array holding the array lists elements and size denotes the number of elementscurrently contained in the array list). The symbolic execution conjoins this condition tothe path condition for the then-path and the negated condition to the path condition ofthe else-path. In this manner all constraints are collected which are needed to deducewhat inputs cause a code path to be taken.

A constraint solver or automatic theorem prover is usually used to decide the feasi-bility of individual execution paths, and to obtain concrete test inputs as representativesof individual execution paths.

Analysis of all paths cannot always be achieved in practice. When loops and recur-sion are present, an unbounded number of code paths may exist. In this case loops andrecursion are usually analyzed only up to a specified number of unfoldings. Even if thenumber of paths is finite, solving the resulting constraint systems is sometimes compu-tationally infeasible, depending on the employed constraint solver or automatic theoremprover.

Symbolic execution in its original form [194] is a static program analysis, since itdoes not actually execute the program but merely analyzes possible execution paths.

6.5 Dynamic Symbolic ExecutionApplying symbolic execution as described above to real-world program is problematic,since such a programs interaction with a stateful environment cannot be forked.

Pex explores the behaviors of a parameterized unit test using a technique called dy-namic symbolic execution [141,67]. This test generation technique consists in execut-ing the program, starting with very simple inputs, while simultaneously performing asingle-path symbolic execution to collect symbolic constraints on the inputs obtainedfrom predicates in branch statements along the execution, and then using a constraintsolver to infer variants of the previous inputs in order to steer future program execu-tions along alternative program paths. In this way, all program paths will be exercised


Step 0 Set J := (intuitively, J is the set of already ana-lyzed program inputs)

Step 1 Choose program input i / J (stop if no such i can be found)Step 2 Output iStep 3 Execute P(i); record path condition C (in particular, C (i) holds)Step 4 Set J := J C (viewing C as the set {i | C (i)})Step 5 Goto Step 1.

Algorithm 6.1. Dynamic symbolic execution

eventually. Operations that are implemented by the external environment are not trackedsymbolically; instead, the actually observed input/output values become part of the con-straints.

As a result, dynamic symbolic execution extends static symbolic execution [194]with additional information that is collected at runtime, which makes the analysis moreprecise [141,139]. By continuation, all analyzed execution paths are feasible, whichavoids the problem of spurious error reports common to overly conservative static anal-ysis techniques.

While additional information is collected on the level of individual execution traceswhich characterize individual execution paths, knowing the structure of the programstill enables the analysis of many execution paths at once, [141,11,49].

Algorithm 6.1 shows the general dynamic symbolic execution algorithm. The choiceof the new inputs in Step 1 decides in which order the different execution paths ofthe program are visited. Pex uses several heuristics that take into account the structureof the program and the already covered branches when deciding on the next programinputs. The goal of Pex strategy is to achieve high statement coverage fast. As an effect,the user just has to set a time limit or another rough analysis bound. Most other tools([141,139,141,67]) explore the execution paths in a fixed search order, and they requirethe user to define detailed bounds on the size and structure of the input data.

6.6 Example

In the following, we explain how Pex handles Listing 5.9, starting from the methodComplicated.

In order to run the code for the first time, Pex needs to supply some argument valuesto Complicated, for example x = 572 and y = 152 (arbitrary values). Pex monitorsthe execution of the code, following the execution into and out of method calls. Withrandomly chosen values, it is very unlikely that we will trigger the rare exception. Pexremembers that we did not cover the throw statement. Pex also remembers all condi-tions that were evaluated; here, it remembers that x = (100 + y) 567%2347.

Knowing that we have not yet covered the throw statement, Pex looks at the pro-gram to find the branch that guards that statement. Last time, we had x = (100+y)567mod 2347. So in order reach the other branch of the if-statement, Pex builds thenegated condition, x = (100+ y) 567 mod 2347, and hands it to a constraint solver.In this case, it is quite simple to solve the constraint system, since one just has to supplyany value for y to compute x.


Pex runs the code again for with the new inputs, say x = (100 + 152) 567mod 2347 and y = 152. This time the throw statement will be covered. Since allstatements have been covered now, Pex will stop.

7 Understanding Pex (Advanced)This section explains various aspects of the tool Pex, that is used to craft inputs to theparameterized unit tests.

7.1 Why Dynamic Symbolic ExecutionSymbolic execution was originally proposed [194] as a static program analysis tech-nique, which is an analysis that only considered the source code of the analyzed pro-gram. This approach works well as long as all decisions can be made on basis of thesource code alone. It becomes problematic when the program contains constructs thatmake reasoning hard (for example accessing memory through arbitrary pointers), andwhen parts of the program are actually unknown (for example when the program com-municates with the environment, for which no source code is available, and whose be-havior has not been specified).

Many .NET programs use unsafe .NET features such as arbitrary memory accessesthrough pointers for performance reasons, and most .NET programs interact with otherunmanaged (non-.NET) components for legacy reasons.

While static symbolic execution algorithms do not have or use any information aboutthe environment into which the program is embedded, dynamic symbolic executiondoes leverage dynamic information that it observes during concrete program executions(information about actually taken execution paths, and the data that is passed aroundbetween the analyzed program and the environment).

Knowing the actually taken execution paths allows Pex to prune the search space.When the program communicates with the environment, Pex builds a model of the en-vironment from the actual data that the environment receives and returns. This modelis an under-approximation of the environment, since Pex does not know the conditionsunder which the environment produces its output. The resulting constraint systems thatPex builds may no longer accurately characterize the programs behavior, and as a con-sequence Pex prunes such paths. Thus, Pex always maintains an under-approximationof the programs behavior, which is appropriate for testing.

Exercise 11. Understanding the Path Condition

Part 1: Solving constraints of Listing 5.9 Call Listing 5.9 from a parameterized unittest, and inspect which values Pex uses as test inputs. Do the values match your expec-tations? Would another set of values be more appropriate for testing? If so why, if not,why not?

Part 2: Playing with Listing 5.9 We revisit Listing 5.9 and see how Pex handles thisexample.1. Add a new test in the project, copy the source of Listing 5.9 and turn it into a

parameterized test.


2. One way to gain more insight on what is happening is the GetPathCondition-String method of PexSymbolicValue class that returns the current path condition.The following code will add a column to the parameter table and fill it in with thepath condition.

i n t Compl i ca t ed ( i n t x , i n t y ) {i f ( x == O b f u s c a t e ( y ) )

throw new R a r e E x c e p t i o n ( ) ;

/ / l o g g i n g t h e p a t h c o n d i t i o ns t r i n g pc = PexSymbol icValue . G e t P a t h C o n d i t i o n S t r i n g ( ) ;PexObserve . ValueForViewing ( " pc " , pc ) ;

r e t u r n 0 ;}

If you run the above code as part of a parameterized unit test, you will notice thatthe path condition is only added to the table when no exception is th

parameterized unit testing with pex tutorial.pdf

Documents

parameterized unit tests

unit tests byhand

generation of good unit

pex tool

pex version

generated test

introductionunit tests

test execution frameworks