advances in unit testing: theory and practice

Advances in Unit Testing: Theory and Practice

Tao Xie1 Nikolai Tillmann2 Pratap Lakshman 2

1 University of Illinois at Urbana-Champaign2 Microsoft Tools for Software Engineers/Developer

Division Materials: http://taoxie.cs.illinois.edu/courses/testing/

2

What is a Unit Test?A unit test is a small program with assertions.

[TestMethod] public void Add(HashSet s,int x, int y){ Assume.IsTrue(!s.IsMember(x)); int oldCount = s.GetSize(); s.Add(x); Assert.AreEqual(set.Count, oldCount + 1);}

Many developers write such unit tests by hand. This involves

determining a meaningful sequence of method calls, selecting exemplary argument values (the test

inputs), stating assertions.

Unit Testing: Benefits Design and specification

by example Code coverage and regression testing

confidence in correctness preserving behavior

Short feedback loop unit tests exercise little code failures are easy to debug

Documentation

Unit Testing: Measuring Quality

Coverage: Are all parts of the program exercised? statements basic blocks explicit/implicit branches …

Assertions: Does the program do the right thing? test oracle

Experience: Just high coverage or large number of assertions

is no good quality indicator. Only both together are!

5

Advantages of tests as specs

Concrete, easy to understand Don’t need new language Easy to see if program meets the

spec Making tests forces you to talk to

customer and learn the problem Making tests forces you to think

about design of system (classes, methods, etc.)

6

Disadvantages of tests as specs Too specific Hard to test that something can’t

happen Can’t withdraw more money than you

have in the system Can’t break into the system Can’t cause a very long transaction that

hangs the system Tends to be verbose

Parameterized Unit Test

Parameterized Unit Test A parameterized unit test is a small

program that takes some inputs and states assumptions and assertions.

7


Parameterized Unit Testing

Parameterized Unit Tests serve as specifications can be leveraged by (automatic) test input

generators fit in development environment, evolve with the

code

Hand-written

Test Generation Process

9

// FooTest.cs[TestClass, PexClass]partial class FooTest{ [PexMethod] void Test(Foo foo) {…}

// FooTest.Test.cspartial class FooTest{ [TestMethod] void Test_1() { this.Test(new Foo(1)); }

[TestMethod] void Test_1() { this.Test(new Foo(2)); } …}

Pex

• User writes parameterized tests• Lives inside a test class • Generated unit tests

• Pex not required for re-execution• xUnit unit tests

xUnit AttributesPex Attributes


Partial Class

Generated

http://msdn.microsoft.com/en-us/library/wa80x488(VS.80).aspx



PUTs separate concerns

PUTs separate two concerns:(1) The specification of external behavior (i.e., assertions)(2) The selection of internal test inputs

(i.e., coverage)

In many cases, a test generation tool (e.g., Pex) can construct a small test suite with high coverage !

PUTs are algebraic specs

A PUT can be read as a universally quantified, conditional axiom.

int name, int data.name ≠ null ⋀ data ≠ null ⇒ equals(

ReadResource(name,WriteResource(name,

data)),data)

Introduction to Software Testing (Ch 1) © Ammann & Offutt

12

Parameterized Unit Tests in JUnit

import org.junit.*;import org.junit.runner.RunWith;import org.junit.runners.Parameterized;import org.junit.runners.Parameterized.Parameters;import static org.junit.Assert.*;import java.util.*;

@RunWith(Parameterized.class)public class ParamTest { public int sum, a, b; public ParamTest (int sum, int a, int b) { this.sum = sum; this.a = a; this.b = b; }

@Parameters public static Collection<Object[]> parameters() { return Arrays.asList (new Object [][] {{0, 0, 0}, {2, 1, 1}}); }

@Test public void additionTest() { assertEquals(sum, a+b); }}

Introduction to Software Testing (Ch 1) © Ammann & Offutt13

JUnit Theories These Are Unit Tests With Actual Parameters

So Far, We’ve Only Seen Parameterless Test Methods Contract Model: Assume, Act, Assert

Assumptions (Preconditions) Limit Values Appropriately Action Performs Activity Under Scrutiny Assertions (Postconditions) Check Result

@Theory public void removeThenAddDoesNotChangeSet( Set<String> set, String string) { // Parameters! assumeTrue(set.contains(string)) ; // Assume Set<String> copy = new HashSet<String>(set); // Act copy.remove(string); copy.add(string); assertTrue (set.equals(copy)); // Assert // // System.out.println(“Instantiated test: “ + set + “, “ + string);}


14

Question: Where Does Data Come From?

Answer: All Combinations of Values from @DataPoint

Annotations Where Assume Clause is True Four (of Nine) Combinations in This Particular

Case Note: @DataPoint Format is an Array.

@DataPoints public static String[] string = {"ant", "bat", "cat"};

@DataPoints public static Set[] sets = { new HashSet(Arrays.asList("ant", "bat")), new HashSet(Arrays.asList(“bat", “cat", “dog“, “elk”)), new HashSet(Arrays.asList(“Snap”, “Crackle”, “Pop")) };


15

JUnit Theories Need BoilerPlateimport org.junit.*;import org.junit.runner.RunWith;import static org.junit.Assert.*;import static org.junit.Assume.*;

import org.junit.experimental.theories.DataPoint;import org.junit.experimental.theories.DataPoints;import org.junit.experimental.theories.Theories;import org.junit.experimental.theories.Theory;

import java.util.*;

@RunWith(Theories.class)public class SetTheoryTest { … // See Earlier Slides}

(Automated) Test Generation

Human Expensive, incomplete, …

Brute Force Pairwise, predefined data, etc…

Tool Automation!!

State-of-the-Art/Practice Test Generation Tools

Running Symbolic PathFinder ...…=============================

========================= results

no errors detected=============================

========================= statistics

elapsed time: 0:00:02states: new=4, visited=0,

backtracked=4, end=2search: maxDepth=3, constraints=0choice generators: thread=1, data=2heap: gc=3, new=271, free=22instructions: 2875max memory: 81MBloaded code: classes=71, methods=884

…

17

http://research.microsoft.com/pex/

Pex is Part of Visual Studio 2015 Enterprise Edition! As new feature of “IntelliTest”

https://www.visualstudio.com/news/vs2015-vs#Testing



Pex4Fun – Turning Pex Online

1,767,012 clicked 'Ask Pex!'

http://pex4fun.com/default.aspx?language=CSharp&sample=_Template



Code Hunt Programming Game

130,000 usersas of March 2015 after 1 year launch

https://www.codehunt.com/




What are Behind Pex

NOT Random: Cheap, Fast “It passed a thousand tests” feeling

…But Dynamic Symbolic Execution:

e.g., Pex, CUTE,EXE White box Constraint Solving

http://research.microsoft.com/pex/

Dynamic Symbolic Execution (DSE) aka. Concolic Testing [Godefroid et al. 05][Sen et al. 05][Tillmann et al. 08]

Instrument code to explore feasible paths

Behind the Scene of Pex

Dynamic Symbolic Execution

Code to generate inputs for:

Constraints to solve

a!=null a!=null &&a.Length>0

a!=null &&a.Length>0 &&a[0]==1234567890

void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

Observed constraints

a==nulla!=null &&!(a.Length>0)a!=null &&a.Length>0 &&a[0]!=1234567890

a!=null &&a.Length>0 &&a[0]==1234567890

Datanull

{}

{0}

{123…}a==null

a.Length>0

a[0]==123…T

TF

T

F

F

Execute&MonitorSolveChoose next path

Done: There is no path left.

Negated condition

There are decision procedures for individual path conditions, but… Number of potential paths grows

exponentially with number of branches Reachable code not known initially Without guidance, same loop might be

unfolded forever

Fitnex search strategy [Xie et al. DSN 09]

Explosion of Search Space

http://taoxie.cs.illinois.edu/publications/dsn09-fitnex.pdf



Successful Case of MSR Testing Tool: Pex & Relatives

Pex (released on May 2008) Shipped with Visual Studio 15 as IntelliTest 30,388 download# (20 months, Feb 08-Oct 09) 22,466 download# (10 months, Apr 13-Jan 14):

Code Digger Active user community: 1,436 forum posts

during ~3 years (Oct 08- Nov 11) Moles (released on Sept 2009)

Shipped with Visual Studio 12 as Fakes “Provide Microsoft Fakes w/ all Visual Studio

editions” got 1,457 community votes

A Journey of Bringing Automated Unit Test Generation to Practice

Lesson 1. Started as (Evolved) Dream

void TestAdd(ArrayList a, object o) { Assume.IsTrue(a!=null); int i = a.Count; a.Add(o); Assert.IsTrue(a[i] == o);}

Parameterized Unit Tests Supported by Pex

Moles/Fakes

Code Digger

Pex4Fun/Code Hunt

Surrounding (Moles/Fakes) Simplifying (Code Digger) Retargeting (Pex4Fun/Code

Hunt)

Lesson 2. Chicken and Egg

Developer/manager: “Who is using your tool?”

Pex team: “Do you want to be the first?” Developer/manager: “I love your tool but

no.”

Tool Adoption by (Mass) Target Users

Tool Shipping with Visual Studio

Macro Perspective

Micro Perspective

Lesson 3. Human Factors – Generated Data Consumed by Human

Developer: “Code digger generates a lot of “\0” strings as input. I can’t find a way to create such a string via my own C# code. Could any one show me a C# snippet? I meant zero terminated string.”

Pex team: “In C#, a \0 in a string does not mean zero-termination. It’s just yet another character in the string (a very simple character where all bits are zero), and you can create as Pex shows the value: “\0”.”

Developer: “Your tool generated “\0”” Pex team: “What did you expect?” Developer: “Marc.”

Lesson 3. Human Factors – Generated Name Consumed by Human

Developer: “Your tool generated a test called Foo001. I don’t like it.”

Pex team: “What did you expect?” Developer:“Foo_Should_Fail_When_Bar_Is_Ne

gative.”

Lesson 3. Human Factors – Generated Results Consumed by Human

Object Creation messages suppressed (related to Covana by Xiao et al. [ICSE’11])

Exception Tree View Exploration Tree

View

Exploration Results View

http://taoxie.cs.illinois.edu/publications/icse11-covana.pdf




Lesson 4. Best vs. Worst Casespublic bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++)

if (y[i] == 15) x++;

if (x == 110) return true;

} return false;}

Key observations: with respect to the coverage target not all paths are equally promising

for branch-node flipping not all branch nodes are equally

promising to flip• Our solution: – Prefer to flip branch nodes on the most promising paths

– Prefer to flip the most promising branch nodes on paths

– Fitness function to measure “promising” extents

Fitnex by Xie et al. [DSN’09]

To avoid local optimal or biases, the fitness-guided strategy is integrated with Pex’s fairness search strategies





Lesson 5. Tool Users’ Stereotypical Mindset or Habits

“Simply one mouse click and then everything would work just perfectly” Often need environment isolation w/ Moles/Fakes or

factory methods, … “One mouse click, a test generation tool would

detect all or most kinds of faults in the code under test” Developer: “Your tool only finds null references.” Pex team: “Did you write any assertions?” Developer: “Assertion???”

“I do not need test generation; I already practice unit testing (and/or TDD). Test generation does not fit into the TDD process”

Lesson 6. Practitioners’ Voice

Gathered feedback from target tool users Directly, e.g., via

MSDN Pex forum, tech support, outreach to MS engineers and .NET user groups

Indirectly, e.g., via interactions with MS Visual Studio team (a tool

vendor to its huge user base) Motivations of Moles

Refactoring testability issue faced resistance in practice

Observation at Agile 2008: high attention on mock objects and tool supports

Lesson 7. Collaboration w/ Academia Win-win collaboration model

Win (Ind Lab): longer-term research innovation, man power, research impacts, …

Win (Univ): powerful infrastructure, relevant/important problems in practice, both research and industry impacts, …

Industry-located Collaborations Faculty visits, e.g., Fitnex, Pex4Fun Student internships, e.g., FloPSy, DyGen,

state cov Academia-located Collaborations

Lesson 7. Collaboration w/ Academia

Academia-located Collaborations Immediate indirect impacts, e.g.,

Reggae [ASE’09s] Rex MSeqGen [FSE’09] DyGen Guided Cov [ICSM’10] state coverage

Long-term indirect impacts, e.g., DySy by Csallner et al. [ICSE’08] Seeker [OOPSLA’11] Covana [ICSE’11]

Summary of Lessons Learned

Pex practice impacts Moles/Fakes, Code Digger, Pex4Fun/Code

Hunt Lessons in transferring tools

Started as (Evolved) Dream Chicken and Egg Human Factors Best vs. Worst Cases Tool Users’ Stereotypical Mindset or Habits Practitioners’ Voice Collaboration w/ Academia

Experience Reports on Successful Tool Transfer

Nikolai Tillmann, Jonathan de Halleux, and Tao Xie. Transferring an Automated Test Generation Tool to Practice: From Pex to Fakes and Code Digger. In Proceedings of ASE 2014, Experience Papers. http://taoxie.cs.illinois.edu/publications/ase14-pexexperiences.pdf

Jian-Guang Lou, Qingwei Lin, Rui Ding, Qiang Fu, Dongmei Zhang, and Tao Xie. Software Analytics for Incident Management of Online Services: An Experience Report. In Proceedings ASE 2013, Experience Paper. http://taoxie.cs.illinois.edu/publications/ase13-sas.pdf

Dongmei Zhang, Shi Han, Yingnong Dang, Jian-Guang Lou, Haidong Zhang, and Tao Xie. Software Analytics in Practice. IEEE Software, Special Issue on the Many Faces of Software Analytics, 2013. http://taoxie.cs.illinois.edu/publications/ieeesoft13-softanalytics.pdf

Yingnong Dang, Dongmei Zhang, Song Ge, Chengyun Chu, Yingjun Qiu, and Tao Xie. XIAO: Tuning Code Clones at Hands of Engineers in Practice. In Proceedings of ACSAC 2012. http://taoxie.cs.illinois.edu/publications/acsac12-xiao.pdf

http://taoxie.cs.illinois.edu/publications/ase14-pexexperiences.pdf

http://taoxie.cs.illinois.edu/publications/ase14-pexexperiences.pdf

http://taoxie.cs.illinois.edu/publications/ase13-sas.pdf

http://taoxie.cs.illinois.edu/publications/ase13-sas.pdf

http://taoxie.cs.illinois.edu/publications/ieeesoft13-softanalytics.pdf

http://taoxie.cs.illinois.edu/publications/ieeesoft13-softanalytics.pdf

http://taoxie.cs.illinois.edu/publications/acsac12-xiao.pdf

http://taoxie.cs.illinois.edu/publications/acsac12-xiao.pdf

39

Parameterized Unit Tests Supported by Pex/Pex4Funusing System;using Microsoft.Pex.Framework;using Microsoft.Pex.Framework.Settings;

[PexClass]public class Set { [PexMethod] public static void testMemberAfterInsertNotEqual(Set s, int i, int j) { PexAssume.IsTrue(s != null); PexAssume.IsTrue(i != j); bool exist = s.member(i); s.insert(j); PexAssert.IsTrue(exist); } ….}

40

Interface for IntSetClass IntSet { public IntSet() {…}; public void insert(int e) { … } public Bool member(int e) { … } public void remove(int e) { … }}

sort IntSet imports Int, Bool signatures new : -> IntSet insert : IntSet × Int -> IntSet member : IntSet × Int -> Bool remove : IntSet × Int -> IntSet

http://www.cs.unc.edu/~stotts/723/adt.html

41

(Buggy) Implementation for IntSet

Class IntSet { public IntSet() {…}; public void insert(int e) { … } public Bool member(int e) { … } public void remove(int e) { … }}See the Set.cs that can be downloaded fromhttp://taoxie.cs.illinois.edu/courses/testing/Set.cs

Let’s copy it to http://pex4fun.com/default.aspx?language=CSharp&sample=_Template And Click “Ask Pex”

http://taoxie.cs.illinois.edu/courses/testing/Set.cs

http://taoxie.cs.illinois.edu/courses/testing/Set.cs



42

Parameterized Unit Tests Supported by Pex/Pex4Funusing System;using Microsoft.Pex.Framework;using Microsoft.Pex.Framework.Settings;

[PexClass]public class Set { [PexMethod] public static void testMemberAfterInsertNotEqual(Set s, int i, int j) { PexAssume.IsTrue(s != null); PexAssume.IsTrue(i != j); bool existOld = s.member(i); s.insert(j); bool exist = s.member(i); PexAssert.IsTrue(existOld == exist); } ….}

Pex4Fun supports only one PexMethod at a time; you can write multiple PexMethods but comment out other lines of “[PexMethod]” except one

43

Axioms for IntSet

variables i, j : Int; s : IntSet

Axioms:member(new(), i) = false member(insert(s, j), i) = if i = j then true else member(s, i)

http://www.cs.unc.edu/~stotts/723/adt.html

Is this complete? How do we know?

44

Guidelines for Completeness

Classify methods: constructors: return IntSet inspectors: take IntSet as argument, returning some

other value. Identify key constructors, capable of

constructing all possible object states e.g., insert, new.

Identify others as auxiliary, e.g., remove is a destructive constructor

Completeness requires (at least): every inspector/auxiliary constructor is defined by

one equation for each key constructor.

45

Add More Axioms

remove(new(), i) = new() remove(insert(s, j), i) =

if i = j then remove(s, i) else insert(remove(s, i), j)

Are we done yet?The completeness criterion (an equation defining member and remove for each of the new and insert constructors) is satisfied.

46

Guidelines for Completeness

But does this really specify sets? Do the following properties hold?

Order of insertion is irrelevant. insert(insert(s, i), j) = insert(insert(s, j), i)

Multiple insertion is irrelevant. insert(insert(s, i), i) = insert(s, i)

47

Interface (Implementation) for UIntStackClass UIntStack { public UIntStack() {…}; public void Push(int k) { … } public void Pop() { … } public int Top() { … } public bool IsEmpty() { … } public int MaxSize() { … } public bool IsMember(int k) { … } public bool Equals(UIntStack s) { … } public int GetNumberOfElements() { … } public bool IsFull() { … }}

See the UIntStack.cs that can be downloaded fromhttp://taoxie.cs.illinois.edu/courses/testing/UIntStack.cs

http://taoxie.cs.illinois.edu/courses/testing/UIntStack.cs


48

Take-Home Exercise: Write Parameterized Unit Tests (PUTs)Class UIntStack { public UIntStack() {…}; public void Push(int k) { … } public void Pop() { … } public int Top() { … } public bool IsEmpty() { … } public int MaxSize() { … } public bool IsMember(int k) { … } public bool Equals(UIntStack s) { … } public int GetNumberOfElements() { … } public bool IsFull() { … }}

Let’s copy it to http://pex4fun.com/default.aspx?language=CSharp&sample=_Template And Click “Ask Pex”

Reminder: you have to comment earlier written “[PexMethod]” before you try Pex on your current PUT (Pex4Fun can handle only one PUT at a time)

See the UIntStack.cs that can be downloaded fromhttp://taoxie.cs.illinois.edu/courses/testing/UIntStack.cs





49

Recall: Parameterized Unit Tests Supported by Pex/Pex4Funusing System;using Microsoft.Pex.Framework;using Microsoft.Pex.Framework.Settings;

[PexClass]public class Set { [PexMethod] public static void testMemberAfterInsertNotEqual(Set s, int i, int j) { PexAssume.IsTrue(s != null); PexAssume.IsTrue(i != j); bool existOld = s.member(i); s.insert(j); bool exist = s.member(i); PexAssert.IsTrue(existOld == exist); } ….}

50

Force Pex/Pex4Fun to Display All Explored Test Inputs/Pathsusing System;using Microsoft.Pex.Framework;using Microsoft.Pex.Framework.Settings;

[PexClass]public class Set { [PexMethod(TestEmissionFilter=PexTestEmissionFilter.All)] public static void testMemberAfterInsertNotEqual(Set s, int i, int j) { PexAssume.IsTrue(s != null); PexAssume.IsTrue(i != j); bool exist = s.member(i); s.insert(j); PexAssert.IsTrue(exist); } ….}

51

Factory Method: Help Pex Generate Desirable Object StatesIn class, we show the factory method as below automatically synthesized by Pex after a user clicks “1 Object Creation” issue and then click “Accept/Edit Factory Method”. But it is not good enough to generate various types of object states.

[PexFactoryMethod(typeof(UIntStack))] public static UIntStack Create(int k_i) { UIntStack uIntStack = new UIntStack(); uIntStack.Push(k_i); return uIntStack;

// TODO: Edit factory method of UIntStack // This method should be able to configure the object in all possible ways. // Add as many parameters as needed, // and assign their values to each field by using the API. }

52

Factory Method: Help Pex Generate Desirable Object StatesBelow is a manually edited/created good factory method to guide Pex to generate various types of object states. Note that Pex also generates argument values for the factory method.

[PexFactoryMethod(typeof(UIntStack))] public static UIntStack CreateVariedSizeAnyElemsStack(int[] elems) { PexAssume.IsNotNull(elems); UIntStack s = new UIntStack(); PexAssume.IsTrue(elems.Length <= (s.MaxSize() + 1));

for (int i = 0; i < elems.Length; i++) s.Push(elems[i]); return s; }

53

One Sample PUTBelow is a manually edited/created good factory method to guide Pex to generate various types of object states. Note that Pex also generates argument values for the factory method.

[PexMethod] public void TestPush([PexAssumeUnderTest]UIntStack s, int i) { //UIntStack s = new UIntStack(); PexAssume.IsTrue(!s.IsMember(i)); int oldCount = s.GetNumberOfElements(); s.Push(i); PexAssert.IsTrue(s.Top() == i); PexAssert.IsTrue(s.GetNumberOfElements() == oldCount+1); PexAssert.IsFalse(s.IsEmpty()); }

54

Pex4Fun Not Supporting Factory Method - WorkaroundIf you try PUTs on Pex4Fun, which doesn’t support factory method, you can “embed” the factory method like the highlighted code portion below

[PexMethod] public void TestPush(int[] elems, int i) { PexAssume.IsNotNull(elems); UIntStack s = new UIntStack(); PexAssume.IsTrue(elems.Length <= (s.MaxSize() + 1)); for (int i = 0; i < elems.Length; i++) s.Push(elems[i]);

//UIntStack s = new UIntStack(); PexAssume.IsTrue(!s.IsMember(i)); int oldCount = s.GetNumberOfElements(); s.Push(i); PexAssert.IsTrue(s.Top() == i); PexAssert.IsTrue(s.GetNumberOfElements() == oldCount+1); PexAssert.IsFalse(s.IsEmpty()); }

Guideline of Writing PUT• Setup: basic set up for invoking the

method under test• Checkpoint: Run Pex to make sure that

you don't miss any Pex assumptions (preconditions) for the PUT

• Assert: add assertions for asserting behavior of the method under test, involving

• Adding Pex assertions• Adding Pex assumptions for helping assert• Adding method sequences for helping

assert

Setup• Select your method under test m• Put its method call in your PUT• Create a parameter for your PUT as

the class under test c (annotated it with [PexAssumeUnderTest])

• Create other parameters for your PUT for parameters of m if any

• Add Pex assumptions for preconditions for all these parameters of PUT if any

Setup - Example[PexMethod] public void TestPush([PexAssumeUnderTest]UIntStack s, int i) { s.Push(i); }

You may write your factory method to help Pex in test generationIf you get exceptions thrown • if indicating program faults, fix them• If indicating lack of PUT assumptions, add PUT assumptions• If indicating insufficient factory method assumptions or

inappropriate scenarios, add PUT assumptions or improve factory method.

Assert• Think about how you can assert the

behavior• Do you need to invoke other (observer)

helper methods in your assertions (besides asserting return values)?

• Do you need to add assumptions so that your assertions can be valid?

• Do you need to add some method sequence before the method under test to set up desirable state and cache values to be used in the assertions?

Targets for Asserting

• Return value of the method under test (MUT)

• Argument object of MUT• Receiver object properties being

modified by MUT (if public fields, directly assertable)

• How to assert them? • Think about the intended behavior!• If you couldn't do so easily, follow the

guidelines discussed next

Cached Public Property Value

• A property value before invoking MUT may need to be cached and later used.

Pattern 2.1/2.2: Assume, Arrange, Act, Assert[PexMethod]void AssumeActAssert(ArrayList list, object item) {

PexAssume.IsNotNull(list); // assumevar count = list.Count; // arrangelist.Add(item); // actAssert.IsTrue(list.Count == count + 1); // assert

}

Argument of MUT

• Argument value of MUT may be usedPattern 2.3:Constructor Test[PexMethod]void Constructor(int capacity) {

var list = new ArrayList(capacity); // createAssertInvariant(list); // assert invariantAssert.AreEqual(capacity, list.Capacity); // assert

}

Reciever or Argument of Earlier Method • Receiver or argument value of a

method before invoking MUTPattern 2.4/5:Roundtrip[PexMethod]void ToStringParseRoundtrip(int value) {

// two-way roundtripstring s = value.ToString();int parsed = int.Parse(s);// assertAssert.AreEqual(value, parsed);

}

value s parsed

Observer Methods• Invoking observer methods on the

modified object statePattern 2.6: State Relation[PexMethod]void InsertContains(string value) {

var list = new List<string>();list.Add(value);Assert.IsTrue(list.Contains(value));

}

Each modified object property should be read by at least one observer method.

Observer Methods cont.• Forcing observer methods to return

specific values (e.g., true or false) can force you to add specific assumptions or scenarios

[PexMethod]void PushIsFull([PexAssumeUnderTest]UIntStack s, int value) { PexAssume.IsTrue(s.GetSize() == (s.GetMaxSize()-1));

s.Push (value);Assert.IsTrue(s.IsFull ());

}

Alternative Computation• Invoking another method/method

sequence to produce a value to be used

Pattern 2.7: Commutative Diagram[PexMethod]void CommutativeDiagram1(int x, int y) { // compute result in one way string z1 = Multiply(x, y).ToString(); // compute result in another way string z2 = Multiply(x.ToString(), y.ToString()); // assert equality if we get here PexAssert.AreEqual(z1, z2);}

Divide and Conquer• Split possible outcomes into cases

(each with pre and post condition)Pattern 2.8: Cases[PexMethod]void BusinessRules(int age, Job job) {

var salary = SalaryManager.ComputeSalary(age, job);PexAssert

.Case(age < 30).Implies(() => salary < 10000)

.Case(job == Job.Manager && age > 35).Implies(() => salary > 10000)

.Case(job == Job.Manager && age < 20).Implies(() => false); }

Class Invariant Checker

• If class invariant checker (repOk) exists or you would be willing to write one, use it to assert

Pattern 2.3:Constructor Test[PexMethod]void Constructor(int capacity) {

var list = new ArrayList(capacity); // createAssertInvariant(list); // assert invariantAssert.AreEqual(capacity, list.Capacity); // assert

}

Other Patterns• Pattern 2.9: Allowed exceptions

• [PexAllowedException(typeof(ArgumentNullException))]• [ExpectedException(typeof(ArgumentNullException))]

• Pattern 2.10: Reachability• [PexExpectedGoals] + throw new PexGoalException();

• Pattern 2.11: Parameterized Stub• No scenarios or assertions

• Pattern 2.12: Input Output Test• void Add(int a, int b, out int result) { … }• int Substract(int a, int b) { … }

• Pattern 2.13/14: Regression Tests• bool Parse(string input) { … }• PexStore.ValueForValidation("result", result);http://research.microsoft.com/en-us/projects/pex/patterns.pdf

http://research.microsoft.com/en-us/projects/pex/patterns.pdf

http://research.microsoft.com/en-us/projects/pex/patterns.pdf

Test-Driven Development (TDD)

Basic Idea: Write tests before code Refine code with new tests

In more detail, TDD is a cycle of steps: Add a test, Run it and watch it fail, Change the code as little as possible such

that the test should pass, Run the test again and see it succeed, Refactor the code if needed.

Note: TDD and specifications

TDD encourages writing specifications before code Exemplary specification

Later, we will generalize TDD toParameterized TDD Axiomatic specifications

Parameterized Test-Driven Development

Write/refine Contract as PUT

Write/refine Code of Implementation

Fix-it (with Pex),Debug with generated tests

Use Generated Tests

for Regression

Run Pex

Bug in PUT

Bug in Code

failures

no failures

Coding Duels

1,767,012 clicked 'Ask Pex!'

Coding Duels

Pex computes “semantic diff” in cloudsecret reference implementation vs.code written in browserYou win when Pex finds no differences

secret

For more info, see our ICSE 2013 SEE paper: http://taoxie.cs.illinois.edu/publications/icse13see-pex4fun.pdf

http://taoxie.cs.illinois.edu/publications/icse13see-pex4fun.pdf

http://taoxie.cs.illinois.edu/publications/icse13see-pex4fun.pdf

Behind the Scene of Pex for Fun

Secret Implementation class Secret {

public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}

Player Implementation class Player {

public static int Puzzle(int x) { return x; }}

class Test {public static void Driver(int x) { if (Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); }}

behaviorSecret Impl == Player Impl

75

1,594,0921,594,092

It’s a game!

iterative gameplayadaptivepersonalizedno cheatingclear winning criterion

secret

code

test cases

Thank you!Questions ?

http://research.microsoft.com/pex

[email protected] Materials: http://taoxie.cs.illinois.edu/courses/testing/

http://research.microsoft.com/pex

mailto:[email protected]