for developer testing
DESCRIPTION
Separation of Concerns. For Developer Testing. Tao Xie Peking University, China North Carolina State University, USA. In collaboration with Nikolai Tillmann , Peli de Halleux , Wolfram Schulte @Microsoft Research and students @NCSU ASE. Background – Separation of Concerns. - PowerPoint PPT PresentationTRANSCRIPT
For Developer Testing
Separation of Concerns
Tao XiePeking University, China
North Carolina State University, USAIn collaboration with Nikolai Tillmann, Peli de Halleux,
Wolfram Schulte @Microsoft Research and students @NCSU ASE
Background – Separation of ConcernsSeparation of Concerns (SoC)
Aspect-Oriented Software Development (AOSD)
“Killer Apps” for SoC/AOSD? Logging, Security Checking, …
Any Other Application Scenarios in Practice? Tool Support for Software Testing?
Software Testing Setup
=?Outputs
Expected
Outputs
Program
+
Test inputs
Test Oracles
Software Testing Problems
=?Outputs
Expected
Outputs
Program
+
Test inputs
Test Oracles
Test Generation Generating high-quality test inputs (e.g.,
achieving high code coverage)
Software Testing Problems
=?Outputs
Expected
Outputs
Program
+
Test inputs
Test Oracles
Test Generation Generating high-quality test inputs (e.g.,
achieving high code coverage)
Test Oracles Specifying high-quality test oracles (e.g.,
guarding against various faults)
The Recipe of Unit/Dev Testing
var list = new ArrayList(); list.Add(item);
Assert.AreEqual(1, list.Count);}
Three essential ingredients: Data Method Sequence Assertionsvoid TestAdd() { int item = 3;
OutlineParameterized Unit Testing
Separate Test Generation and Behavior Specification
Test Generalization Localize Crosscutting Concerns of Behavior Under
TestObject Creation with Factory Method
Localize Crosscutting Concerns of Object CreationEnvironment Isolation with Moles
Localize Crosscutting Concerns of Environment Dependency
Parameterized Unit Testing
void TestAdd(ArrayList list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}
Parameterized Unit Test = Unit Test with Parameters
Separation of concerns Data is generated by a tool Developer can focus on functional
specification
[Tillmann&Schulte ESEC/FSE 05]
Dynamic Symbolic Execution
Code to generate inputs for:
Constraints to solve
a!=null a!=null &&a.Length>0
a!=null &&a.Length>0 &&a[0]==1234567890
void TestMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
Observed constraints
a==nulla!=null &&!(a.Length>0)a!=null &&a.Length>0 &&a[0]!=1234567890
a!=null &&a.Length>0 &&a[0]==1234567890
Datanull
{}
{0}
{123…}a==null
a.Length>0
a[0]==123…T
TF
T
F
F
Execute&MonitorSolveChoose next path
Done: There is no path left.
Negated condition
Pex:Visual Studio Power Tool
Download counts (20 months)(Feb. 2008 - Oct. 2009 )
Academic: 17,366 Devlabs: 13,022 Total: 30,388
http://research.microsoft.com/projects/pex/
Dynamic Symbolic Execution
[Tillmann&de Halleux TAP 08]
Pex for Fun Web-based Learning Tool
669,584 clicked 'Ask Pex!‘ since 2010 summer
Try it at http://www.pexforfun.com/
Parameterized Unit TestingGetting PopularParameterized Unit Tests (PUTs) commonly
supported by various test frameworks .NET: Supported by .NET test frameworks
http://www.mbunit.com/ http://www.nunit.org/ …
Java: Supported by JUnit 4.X http://www.junit.org/
Generating test inputs for PUTs supported by tools .NET: Supported by Microsoft Research Pex
http://research.microsoft.com/Pex/ Java: Supported by Agitar AgitarOne
http://www.agitar.com/
Parameterized vs. Conventional Unit Tests PUTs vs. CUTs
Benefits over CUTs Help describe behaviours for all test arguments
Address two main issues with CUTs Missing test data (that would exercise
important behavior)▪ Low fault-detection capability
Redundant test data and scenarios (that exercises the same behaviour)▪ Redundant unit tests
OutlineParameterized Unit Testing
Separate Test Generation and Behavior Specification
Test Generalization Localize Crosscutting Concerns of Behavior Under
TestObject Creation with Factory Method
Localize Crosscutting Concerns of Object CreationEnvironment Isolation with Moles
Localize Crosscutting Concerns of Environment Dependency
An Example using IntStack
public void CUT1() { int elem = 1; IntStack stk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count());}
Three CUTs
public void CUT2() { int elem = 30; IntStack stk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count());}
public void CUT3() { int elem1 = 1, elem2 = 30; IntStack stk = new IntStack(); stk.Push(elem1); stk.Push(elem2); Assert.AreEqual(2, stk.Count());}
CUT1 and CUT2 exercise push with different test data
CUT3 exercises push when stack is not empty
Two main issues with CUTs: Fault-detection
capability: undetected defect where things go wrong when passing a negative value to push
Redundant tests: CUT2 is redundant with respect to CUT1
Test Generalization: IntStack
public void CUT1() { int elem = 1; IntStack stk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count());}
Three CUTs
public void CUT2() { int elem = 30; IntStack stk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count());}
public void CUT3() { int elem1 = 1, elem2 = 30; IntStack stk = new IntStack(); stk.Push(elem1); stk.Push(elem2); Assert.AreEqual(2, stk.Count());}
No need to describe test data• Generated automatically
Single PUT replaces multiple CUTs• With reduced size of test
code
public void PUT(int[] elem) { Assume.IsTrue(elem != null); IntStack stk = new IntStack(); for(int i in elem) stk.push(elem); Assert.AreEqual(elem.Length, stk.count());}
An equivalent PUT
[Thummalapenta et al. FASE 11]
Ring a Bell? Refactoring Clones?
Refactoring clone instances (crosscutting concerns) to a single copy (aspects) Need pointcuts to specify join points, i.e., which
original code locations to weave the aspects
Test generalization is different Original code locations (old CUTs) are thrown away PUTs + Pex generate new CUTs No need of join points or pointcuts
OutlineParameterized Unit Testing
Separate Test Generation and Behavior Specification
Test Generalization Localize Crosscutting Concerns of Behavior Under
TestObject Creation with Factory Method
Localize Crosscutting Concerns of Object CreationEnvironment Isolation with Moles
Localize Crosscutting Concerns of Environment Dependency
Object Creation
void TestAdd(ArrayList list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}
Object Creation Pex uses public methods to
configure non-public object fields Heuristics built-in to deal with
common types User can help if needed[PexFactoryMethod(typeof(ArrayList))]public static ArrayList Create(int capacity, object[] items) { var list = new ArrayList(capacity); foreach (var item in items) list.Add(item); return list;}
“Join Point” for Factory Method
[PexFactoryMethod(typeof(ArrayList))]public static ArrayList Create(int capacity, object[] items) { var list = new ArrayList(capacity); foreach (var item in items) list.Add(item); return list;}
void TestAdd(ArrayList list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}
OutlineParameterized Unit Testing
Separate Test Generation and Behavior Specification
Test Generalization Localize Crosscutting Concerns of Behavior Under
TestObject Creation with Factory Method
Localize Crosscutting Concerns of Object CreationEnvironment Isolation with Moles
Localize Crosscutting Concerns of Environment Dependency
Unit Testing is Not That Easy
Components depend on other components
Hidden Integration Testsvoid FileExistsTest() { File.Write(“foo.txt”, “”); var result = IsFileEmpty(“foo.txt”) Assert.IsTrue(result);}
bool IsFileEmpty(string file) { var content = File.ReadAllText(file); return content.Length == 0;}
File.ReadAllText(file);
File.Write(“foo.txt”, “”);
Isolation is Critical Slow, complicated setup, non-deterministic
tests Solution: Replace by Simpler Environment
(“mocking”)
Testable Design: Abstraction layer + Dependency Injection + Mocks for testing Simply use virtual methods (i.e., overridenable)
Hard-coded Design: No abstraction layer, static methods, sealed types (i.e., not inheritable).
Stubs and Moles Framework
Replace any .NET method with a Delegate similar to a C++ function pointer
Method can be overridden? Use Stubs Interfaces Abstract classes Virtual methods in non-sealed types
Method cannot be overridden? Use Moles
Hard-coded Design Existing external components cannot
be re-factored SharePoint, ASP .NET, …
Need mechanism to stub non-virtual methods Static methods, methods in sealed types,
constructors
Moles – Delegate Based Detours Method redirected to user delegate, i.e.
moled
Requires code instrumentation
bool result = IsFileEmpty(“foo.txt”);Assert.IsTrue(result);
MFile.ReadAllTextString = file => “”; expression lambda : (input parameters) => expression
[de Halleux&Tillmann TOOLS 10]
Moles under the Hood
File.ReadAllText(string name) {
}
mscorlib.dll
File.ReadAllText(string name) { var d = GetDetour(); if (d != null) return d();
}
push ecxpush edxpush eax
.NET RuntimeJust In Time
Compiler
ExtendedReflection
Isolated Parameterized Unit Testing Automated White box Analysis does not
‘understand’ the environment
Isolate Code using Stubs and Moles
if (DateTime.Now == new DateTime(2000,1,1)) throw new Y2KException();
DateTime.Now
???
public void TestY2k(DateTime dt) { MDateTime.NowGet = () => dt ...}
MDateTime.NowGet = () => dt DateTime.Now ==
dt
Separation of Concerns for Developer TestingParameterized Unit Testing
Separate Test Generation and Behavior Specification
Test Generalization Localize Crosscutting Concerns of Behavior Under
TestObject Creation with Factory Method
Localize Crosscutting Concerns of Object CreationEnvironment Isolation with Moles
Localize Crosscutting Concerns of Environment DependencyApplication Scenarios of Separation of
Concerns in Practice
Thank you!
Questions ?
https://sites.google.com/site/asergrp/
http://research.microsoft.com/projects/pex/http://
www.pexforfun.com/
http://research.microsoft.com/en-us/projects/pex/community.aspx#publicationshttp://
pexase.codeplex.com/