separation of concerns tao xie peking university, china north carolina state university, usa in...

For Developer Testing

Separation of Concerns

Tao XiePeking University, China

North Carolina State University, USAIn collaboration with Nikolai Tillmann, Peli de Halleux,

Wolfram Schulte @Microsoft Research and students @NCSU ASE

Background – Separation of Concerns

Separation of Concerns (SoC)

Aspect-Oriented Software Development (AOSD)

“Killer Apps” for SoC/AOSD? Logging, Security Checking, …

Any Other Application Scenarios in Practice? Tool Support for Software Testing?

Software Testing Setup

=?

Outputs

Expected

Outputs

Program

+

Test inputs

Test Oracles

Software Testing Problems

=?

Outputs

Expected

Outputs

Program

+

Test inputs

Test Oracles

Test Generation Generating high-quality test inputs (e.g.,

achieving high code coverage)

Software Testing Problems

=?

Outputs

Expected

Outputs

Program

+

Test inputs

Test Oracles

Test Generation Generating high-quality test inputs (e.g.,

achieving high code coverage)

Test Oracles Specifying high-quality test oracles (e.g.,

guarding against various faults)

The Recipe of Unit/Dev Testing

var list = new ArrayList(); list.Add(item);

Assert.AreEqual(1, list.Count);}

Three essential ingredients: Data Method Sequence Assertionsvoid TestAdd() { int item = 3;

Outline

Parameterized Unit Testing Separate Test Generation and Behavior

Specification Test Generalization

Localize Crosscutting Concerns of Behavior Under Test

Object Creation with Factory Method Localize Crosscutting Concerns of Object

Creation Environment Isolation with Moles

Localize Crosscutting Concerns of Environment Dependency

Parameterized Unit Testing

void TestAdd(ArrayList list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}

Parameterized Unit Test = Unit Test with Parameters

Separation of concerns Data is generated by a tool Developer can focus on functional

specification

[Tillmann&Schulte ESEC/FSE 05]

Dynamic Symbolic Execution

Code to generate inputs for:

Constraints to solve

a!=null a!=null &&a.Length>0

a!=null &&a.Length>0 &&a[0]==1234567890

void TestMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

Observed constraints

a==nulla!=null &&!(a.Length>0)a!=null &&a.Length>0 &&a[0]!=1234567890

a!=null &&a.Length>0 &&a[0]==1234567890

Data

null

{}

{0}

{123…}a==null

a.Length>0

a[0]==123…T

TF

T

F

F

Execute&MonitorSolve

Choose next path

Done: There is no path left.

Negated condition

Pex:Visual Studio Power Tool

Download counts (20 months)(Feb. 2008 - Oct. 2009 )

Academic: 17,366 Devlabs: 13,022 Total: 30,388

http://research.microsoft.com/projects/pex/

Dynamic Symbolic Execution

[Tillmann&de Halleux TAP 08]

Pex for Fun Web-based Learning Tool

669,584 clicked 'Ask Pex!‘ since 2010 summer

Try it at http://www.pexforfun.com/

Parameterized Unit TestingGetting PopularParameterized Unit Tests (PUTs) commonly

supported by various test frameworks .NET: Supported by .NET test frameworks

http://www.mbunit.com/ http://www.nunit.org/ …

Java: Supported by JUnit 4.X http://www.junit.org/

Generating test inputs for PUTs supported by tools .NET: Supported by Microsoft Research Pex

http://research.microsoft.com/Pex/ Java: Supported by Agitar AgitarOne

http://www.agitar.com/

Parameterized vs. Conventional Unit Tests PUTs vs. CUTs

Benefits over CUTs Help describe behaviours for all test arguments

Address two main issues with CUTs Missing test data (that would exercise

important behavior)▪ Low fault-detection capability

Redundant test data and scenarios (that exercises the same behaviour)▪ Redundant unit tests

Outline

Parameterized Unit Testing Separate Test Generation and Behavior

Specification Test Generalization

Localize Crosscutting Concerns of Behavior Under Test

Object Creation with Factory Method Localize Crosscutting Concerns of Object

Creation Environment Isolation with Moles

Localize Crosscutting Concerns of Environment Dependency

An Example using IntStack

public void CUT1() { int elem = 1; IntStack stk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count());}

Three CUTs

public void CUT2() { int elem = 30; IntStack stk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count());}

public void CUT3() { int elem1 = 1, elem2 = 30; IntStack stk = new IntStack(); stk.Push(elem1); stk.Push(elem2); Assert.AreEqual(2, stk.Count());}

CUT1 and CUT2 exercise push with different test data

CUT3 exercises push when stack is not empty

Two main issues with CUTs: Fault-detection

capability: undetected defect where things go wrong when passing a negative value to push

Redundant tests: CUT2 is redundant with respect to CUT1

Test Generalization: IntStack

public void CUT1() { int elem = 1; IntStack stk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count());}

Three CUTs

public void CUT2() { int elem = 30; IntStack stk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count());}

public void CUT3() { int elem1 = 1, elem2 = 30; IntStack stk = new IntStack(); stk.Push(elem1); stk.Push(elem2); Assert.AreEqual(2, stk.Count());}

No need to describe test data• Generated automatically

Single PUT replaces multiple CUTs• With reduced size of test

code

public void PUT(int[] elem) { Assume.IsTrue(elem != null); IntStack stk = new IntStack(); for(int i in elem) stk.push(elem); Assert.AreEqual(elem.Length, stk.count());}

An equivalent PUT

[Thummalapenta et al. FASE 11]

Ring a Bell? Refactoring Clones?

Refactoring clone instances (crosscutting concerns) to a single copy (aspects) Need pointcuts to specify join points, i.e., which

original code locations to weave the aspects

Test generalization is different Original code locations (old CUTs) are thrown away PUTs + Pex generate new CUTs No need of join points or pointcuts

Outline

Parameterized Unit Testing Separate Test Generation and Behavior

Specification Test Generalization

Localize Crosscutting Concerns of Behavior Under Test

Object Creation with Factory Method Localize Crosscutting Concerns of Object

Creation Environment Isolation with Moles

Localize Crosscutting Concerns of Environment Dependency

Object Creation

void TestAdd(ArrayList list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}

Object Creation

Pex uses public methods to configure non-public object fields

Heuristics built-in to deal with common types

User can help if needed[PexFactoryMethod(typeof(ArrayList))]public static ArrayList Create(int capacity, object[] items) { var list = new ArrayList(capacity); foreach (var item in items) list.Add(item); return list;}

“Join Point” for Factory Method

[PexFactoryMethod(typeof(ArrayList))]public static ArrayList Create(int capacity, object[] items) { var list = new ArrayList(capacity); foreach (var item in items) list.Add(item); return list;}

void TestAdd(ArrayList list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}

Outline

Parameterized Unit Testing Separate Test Generation and Behavior

Specification Test Generalization

Localize Crosscutting Concerns of Behavior Under Test

Object Creation with Factory Method Localize Crosscutting Concerns of Object

Creation Environment Isolation with Moles

Localize Crosscutting Concerns of Environment Dependency

Unit Testing is Not That Easy

Components depend on other components

Hidden Integration Testsvoid FileExistsTest() { File.Write(“foo.txt”, “”); var result = IsFileEmpty(“foo.txt”) Assert.IsTrue(result);}

bool IsFileEmpty(string file) { var content = File.ReadAllText(file); return content.Length == 0;}

File.ReadAllText(file);

File.Write(“foo.txt”, “”);

Isolation is Critical

Slow, complicated setup, non-deterministic tests

Solution: Replace by Simpler Environment (“mocking”)

Testable Design: Abstraction layer + Dependency Injection + Mocks for testing Simply use virtual methods (i.e., overridenable)

Hard-coded Design: No abstraction layer, static methods, sealed types (i.e., not inheritable).

Stubs and Moles Framework

Replace any .NET method with a Delegate similar to a C++ function pointer

Method can be overridden? Use Stubs Interfaces Abstract classes Virtual methods in non-sealed types

Method cannot be overridden? Use Moles

Hard-coded Design

Existing external components cannot be re-factored SharePoint, ASP .NET, …

Need mechanism to stub non-virtual methods Static methods, methods in sealed types,

constructors

Moles – Delegate Based Detours

Method redirected to user delegate, i.e. moled

Requires code instrumentation

bool result = IsFileEmpty(“foo.txt”);Assert.IsTrue(result);

MFile.ReadAllTextString = file => “”; expression lambda : (input parameters) => expression

[de Halleux&Tillmann TOOLS 10]

Moles under the Hood

File.ReadAllText(string name) {

}

mscorlib.dll

File.ReadAllText(string name) { var d = GetDetour(); if (d != null) return d();

}

push ecxpush edxpush eax

.NET RuntimeJust In Time

Compiler

ExtendedReflection

Isolated Parameterized Unit Testing

Automated White box Analysis does not ‘understand’ the environment

Isolate Code using Stubs and Moles

if (DateTime.Now == new DateTime(2000,1,1)) throw new Y2KException();

DateTime.Now

???

public void TestY2k(DateTime dt) { MDateTime.NowGet = () => dt ...}

MDateTime.NowGet = () => dt DateTime.Now ==

dt

Separation of Concerns for Developer Testing Parameterized Unit Testing

Separate Test Generation and Behavior Specification

Test Generalization Localize Crosscutting Concerns of Behavior

Under Test Object Creation with Factory Method

Localize Crosscutting Concerns of Object Creation

Environment Isolation with Moles Localize Crosscutting Concerns of Environment

DependencyApplication Scenarios of Separation of Concerns in Practice

Thank you!

Questions ?

https://sites.google.com/site/asergrp/

http://research.microsoft.com/projects/pex/http://

www.pexforfun.com/

http://research.microsoft.com/en-us/projects/pex/community.aspx#publicationshttp://

pexase.codeplex.com/