unit testing for performance using ntime - · pdf fileunit testing for performance using ntime...

UNIT TESTING FOR PERFORMANCE USING NTIME

Author:

Ramajayam Ramalingam

TABLE OF CONTENTS

AN INTRODUCTION TO NTIME 2

PERFORMANCE ENGINEERING AND NTIME 3

TESTING FOR APPLICATION PERFORMANCE USING NTIME 4 NTime Test Class Setup 4 NTime Possible Performance Test Scenarios 4 Sample Implementation 5 Setup a .NET Console Application 6 Create Entities/Classes to be Tested 6 Write Performance Test Scripts Using NTime 7 Testing & Seeing Performance Results in NTime 8

CONTINUOUS INTEGRATION—RUNNING NTIME IN MS BUILD 9 Creating NTime Task for MSBuild 9 Integrating NTime Task with MSBuild 9

BENEFITS OF NTIME 11

CONCLUSION 11

REFERENCES 11


NTime is a free, open source, performance benchmarking tool created by Adam Slosarski, to write performance tests for .NET applications.

NTime ensures repeated performance testing of an application at desired intervals. It can test against the duration or number of hits to a method or make use of Windows Performance Monitor probes to check for optimized code solutions. NTime is not a profiler, but a quick way to set time or performance boundaries on .NET applications which are susceptible to performance bottlenecks. In other words, an upfront performance specification/metric is required to implement NTime.NTime’s source code, installer and tutorials are freely available at (http://goo.gl/8fMaz). They can be downloaded and used for measuring and achieving performance goals of any application.

AN INTRODUCTION TO NTIME

2

The performance of a software application is critical in ensuring user satisfaction and meeting service-level agreements (SLAs). Therefore, performance testing plays an important role in the development life cycle. Performance needs to be engineered throughout the software development lifecycle. Performance goals must be defi ned and metrics must be gathered and analyzed at each phase of the development life cycle to ensure that performance goals are being met. This whitepaper introduces NTime, a performance testing tool that helps test performance goals throughout the development process and measure key performance metrics, such as the duration of a method call, number of hits to a method and code optimization levels. This paper also demonstrates the need for NTime and explains how to execute a performance test for .NET applications using this freely available open source tool.

NTime can be used by both system architects and .NET developers.

• Architects Architects identify key scenarios

for performance, translate them to performance goals for each layer and hand over the minimum performance measures of modules to the developers tasked with NTime testing.

• .NET Developers .NET Developers would be interested

in the diverse features offered by NTime for unit testing the performance of their modules, which in turn will help them realize the minimum required performance.

Performance, as a quality of service requirement, has to be engineered and cannot be planned/executed overnight. During different phases of the development life cycle, the following performance principles must be applied:• Gathering Requirements

In this phase, performance objectives are set by the development team after consultations with the customer. Units of measure, such as CPU time and memory footprint, are identified.

• Designing for PerformanceKey performance scenarios, along with the workload, are identified in this phase. Examples of scenarios include meeting a performance goal or addressing a potential threat to the overall performance.

• ImplementationIn this phase, performance requirements are implemented in line with established design guidelines. The performance-verifying unit test cases are created when bottlenecks are expected or a performance scenario’s functionality keeps changing.

• Verification/TestingRegular testing of the code for performance ensures that the final performance goals are met. Various

tools should be used to test the scenarios identified for performance. Metrics collected for the units of measure from these tools are compared with the performance goals.

Performance Engineering and NTimeThe performance goals that are agreed upon in the requirements gathering phase are translated to measurable units, such as resource utilization and response time.

These units are the inputs to the NTime testing scripts. They should be further translated and used as duration, number of hits and performance counters. During the implementation phase, test cases are written in NTime in accordance with the performance specifications for the identified unit test scenarios.

Just like NUnit, NTime can be repeatedly run to verify that, in spite of the changes the developers have made to a particular function, it still runs within the performance limits. NTime, if kept open alongside the IDE, reloads the updated tests automatically, as and when the application is built.

PERFORMANCE ENGINEERING AND NTIME

3Figure 1: NTime GUI


NTime Test Class SetupTo set up the performance test environment before and after the test execution, NTime can be configured using the following attributes: • TimerFixture: Indicates that a class is to undergo

performance testing. • TimerFixtureSetup: Specifies a global set-up

environment for the entire performance test. The content of this block is executed before the first test is started. This is like a constructor for a class.

• TimerFixtureTearDown: Specifies a global tear-down environment for the entire performance test. This block will be executed after all tests are executed/completed. This is like a destructor for a class.

• TimerSetUp: Specifies a place to locally set up the environment for a single test. Before each and every performance testing method starts, this block of code is executed and necessary parts/objects are initialized.

• TimerTearDown: Specifies a place to locally tear down the environment for each performance test. After a test method has finished its execution, contents in this block are executed at the end of each test.

• TimerIgnore: Instructs the performance test tool to skip performance test and mark it as warning information.

• Syntax: Gives an overview of how these attributes should be used with proper syntax.

NTime Possible Performance Test ScenariosUsing NTime, the following performance scenarios for an application can be verified: I. Duration Testing

Duration testing involves testing the time taken for completing a function call. This can be achieved using the TimerDurationTest attribute. For example:

[TimerDurationTest(50, Threads = 10, Unit = TimePeriod.Milliseconds)]public static void searchPerformanceTest() { DemoCustomer customerService = new DemoCustomer(); IEnumerable<Product> searchResult = customerService.searchProducts(“Loan”); foreach (Product product in searchResult) { string result = “ID :” + product.ID; result += “;\nName :” + product.Name+ “, “ + product.Description; Console.WriteLine(result); } }

50 denotes the expected duration of this call. The unit parameter specifies the unit of duration, which could be seconds, milliseconds or minutes. 10 indicates the number of threads being used for this test.

When the test is run in NTime, if the actual duration is 50 milliseconds or less, the test is shown in the NTime UI as Passed, otherwise it is shown as Failed/Rejected.

II. Hit Count TestingHit count testing measures the number of times a method can be run/hit within a specified timeframe. This can be achieved in NTime using the TimerHitCountTest attribute. For example:

[TimerHitCountTest(25, Threads = 3, Unit = TimePeriod.Second)]public static void loginPerformanceTest(){ DemoCustomer customerService = new DemoCustomer(); string result = customerService.makeLogin(); Console.WriteLine(result);}

4

TESTING FOR APPLICATION PERFORMANCE USING NTIME

25 is the expected number of times this method will be invoked in the given time period.

3 indicates the number of threads that NTime should use to call this method.

NTime creates 3 threads and makes calls to this method. If the method has been invoked for 25 times or more within a second, this method is marked as “Passed”, otherwise the method is marked as “Failed”.

III. Counter TestingCounter testing is where the Windows Performance Monitor (WPM) probes can be utilized in NTime tests to measure the system resources consumed by the application.

TimerCounterTest attribute is used to achieve the same.

[TimerCounterTest(“CategoryName”, “CounterName”, InstanceName = “Instance”, Threads = int, MinimumValue = int, MaximumValue = int)]

CategoryName is any of the Performance Objects that one can see in the PerfMon counter (such as Processor or Memory). CounterName is the specific counters within the CategoryName, such as “Pages/sec” or “% Processor Time”.

To gather performance-monitoring statistics, methods need to exercise the TimerCounterTest attribute.

Note that the normal execution of a method call might have to be prolonged to ensure that enough data is gathered. The following example tests are used to ensure CPU utilization rates are within 1 to 70 percent.

[TimerCounterTest(“Process”, “% Processor Time”, Threads = 1, InstanceName = “NTimeGUI”, MinimumValue = 1, MaximumValue = 70)]public static void loadPerformanceTest() {

loginPerformanceTest(); searchPerformanceTest(); }

Sample Implementation Business ScenarioABC Bank is developing an application to handle a large amount of customer data. ABC Bank’s top priority in the application is to achieve maximum performance gains. The application should be able to handle large hits from several users, run fast and utilize the minimum number of resources possible.The application’s performance specifications include the following:1. Application should support 50 concurrent users

signing in per second2. Search module should return results within 2

seconds at a peak load of 10 concurrent searches.3. For any given operation or operations combined, CPU

utilization should not exceed 70% for 1 user

Performance Testing ScenariosThe following sample performance test implementation intends to measure the performance requirements using NTime:1. To determine whether the application meets the

concurrent access load, the number of hits will be measured

2. To determine whether search results are returned within the agreed upon time, the duration of method calls will be measured

3. To determine the CPU utilization, windows performance counters will be measured

5


Implementing NTime: Walkthrough Database SetupCreate a database named BankDB with the following two tables and host it in your Database Server.

Setup a .NET Console Application1. Create a .NET console application2. Add the following ‘dll’ references to your projects: a. NTime.Framework.dll (at folder “…\NTime\

Bin\” for .NET Framework 3.5 and below. For .NET Framework 4.0 and above, use Recompiled version of NTime Source Code for corresponding version).

3. To use the Entity Framework as the data source: a. Select “Add new Items” by right clicking the

project. b. Pick the “ADO.NET Entity Data Model”, name it

“Bank DataModel” and press the Enter button. Consequently, select “Generate from Database” for creating the models.

c. Follow the following sequence to set the data provider:

New Connection -> Press “Change…” Button -> Select “Others” as Data source -> Select “.NET Framework Data Provider for SQL Server” as the data Provider.

d. Type your database server machine name where you have created and saved the database BankDB in the Server Name text box.

e. Use the SQL Server Authentication with the appropriate User Name and Password that can be used to connect to the database server.

f. Select the “BankDB” as the Database and press “Ok” button.

g. In the “Entity Data Model Wizard”, select the “Yes” option and press “Next” button.

h. From the tables listed by expanding the “Tables” node, select the Customer and Product tables and then press the Finish button. Now your

Entity Data Model is ready to be used by the Application.

Create Entities/Classes to be Tested1. Create a Customer Data Service Interface to

interact with the database (through the above Entity Framework). This is called the Data Access Layer.

public interface ICustomerDataService { IEnumerable<Product> SearchProducts(string productName); Customer Login(string userName, string password); } public class CustomerDataService:ICustomerDataService { public IEnumerable<Product> SearchProducts(string productName) { return (from product in (new BankDBEntities()).Products where product.Name.ToLower().Contains(ProductName.ToLower()) select product); } public Customer Login(string userName,

6

Figure 2: Database Bank DB

string password) { try { return (new BankDBEntities()).Customers.Where(cust => cust.UserName == UserName && cust.Password == Password).Single(); } catch (Exception exception) { return null; } } }

2. Create a class (named DemoCustomer) that performs two operations, namely makeLogin and searchProducts as the following:

class DemoCustomer { public string makeLogin() { string result; CustomerDataService service = new CustomerDataService(); Customer loggedCustomer = service.Login(“Ramajayam”, “rama”); if (loggedCustomer == null) result = “Invalid Login Information Entered\n”; else { result = “Log-in Successfull\n”; result += “User Name: “ + loggedCustomer.UserName + “;\nFullName: “ + loggedCustomer.FullName; result += “;\nAddress: “ + loggedCustomer.Location+”\n\n”; } return result; } public IEnumerable<Product> searchProducts(string searchTerm) { searchTerm = searchTerm.ToLower(); CustomerDataService service = new

CustomerDataService(); return service.SearchProducts(searchTerm); } }

Write Performance Test Scripts Using NTime1. Add the NTime.Framework.dll Assembly file as

Reference to your project.2. Write a class which is going to performance test the

above operations as indicated below:

[TimerFixture] public class BankPerformanceTest { //Test if the 50 logins could be achieved within a second, assuming there could be an average of 10 threads. [TimerHitCountTest(50, Threads = 10, Unit = TimePeriod.Second)] public static void loginPerformanceTest() { DemoCustomer customerService = new DemoCustomer(); string result = customerService.makeLogin(); Console.WriteLine(result); } //Test if the search method is executed within 2 seconds for 10 concurrent searches [TimerDurationTest(2, Threads = 10, Unit = TimePeriod.Second)] public static void searchPerformanceTest() { DemoCustomer customerService = new DemoCustomer(); IEnumerable<Product> searchResult=customerService.searchProducts(“Loan”); foreach (Product product in searchResult) { string result = “ID :” + product.ID; result += “;\nName :” + product.Name+ “, “ + product.Description; Console.WriteLine(result); }

7


} // For a combined operation of login and search, the CPU utilization should not be more than 70%

[TimerCounterTest(“Process”, “% Processor Time”, Threads = 1, InstanceName = “NTimeGUI”, MinimumValue = 1, MaximumValue = 70)] public static void loadPerformanceTest() { loginPerformanceTest(); searchPerformanceTest(); } }

3. Now compile and run the project.4. The application is now ready for performance

testing using the NTime tool.

Testing & Seeing Performance Results in NTimeThe procedure for performance testing on the application using NTime is as follows:1. Open the NTime.exe file and click the “Add

Reference” option. Then browse and select the output Assembly file of the above application.

2. After adding the application’s Assembly file into the NTime tool, press the “START” button/option shown in the NTime UI.

3. The performance test results will be displayed, indicating whether the methods were accepted or rejected based on the performance specification of the test methods.

4. If a performance test method is rejected, then refactor the method until it achieves the specified performance level.

5. Only after all the performance test methods are accepted will the application have met the agreed upon performance specification.

The following snapshot shows the results in the NTime UI:

8

Figure 3: Snapshot of NTime Test Result

NTime does not have any built-in support for running through automated builds, requiring you to write your own task to be used in the MSBuild tool. Once you create the task, it can be integrated into MSBuild.

Creating NTime Task for MSBuildThe following steps show how to create a Task for NTime that will be used in the MSBuild:1. Create a Class Library Project in Visual Studio2. Add the following Assemblies to this project.2.1. NTime.Framework.dll2.2. Microsoft.Build.Utilies.dll (Version: 4.0)2.3. Microsoft.Build.Framework.dll3. Create a class that inherits the Parent Class “Task”

and override the Execute() method:

public class NTime:Task { public override bool Execute() { if (AssemblyFullPathValue == null) { Console.WriteLine(“\nAssembly Name must be entered\n”); return false; } Program p = new Program(); p.Domain = new AssemblyDomain(); p.Open(AssemblyFullPathValue); if (p.InValidAssembly == true) { Console.WriteLine(“”); return false; } if (p.rl.HasTimerFixture == false) { Console.WriteLine(“No TimerFixture attributed Class found “); return false; }

p.Run(); if(StopOnRejectedTest==true) if (p.rl.RejectedTests != 0) return false; Console.WriteLine(“”); return true; } [Required] public string AssemblyFullPathValue { get; set; } public bool StopOnRejectedTest { get; set; } }

Integrating NTime Task with MSBuildFollow these steps to integrate the NTime task with MSBuild: 1. Compile the above Ntime task into NTimetask.dll.

Copy NTime.Framework.dll and NTimeTask.dll into a specific folder, viz. <<NTIME_LIB>>.

2. Write a Build file that will be input to MSBuild.exe as the following:

<Project DefaultTargets=”NTimeTarget” xmlns=”http://schemas.microsoft.com/

developer/msbuild/2003”> <UsingTask AssemblyFile=”<<NTIME_LIB>>\

NTimeTask.dll” TaskName=”NTimeTask.NTime”/>

<Target Name=”NTimeTarget”> <Message Text=”Starting the Performance

Testing...”/> <NTime AssemblyFullPathValue=”You

rAssemblyPath1\AssemblyName1.dll” ContinueOnError=”true”/>

<Message Text=”Performance Testing Ended...”/> <OnError ExecuteTargets=”ErrorTarget”/> </Target> <Target Name=”ErrorTarget”> <Message Text=”One Of the Build Tasks had

Failed.”/> </Target> </Project>

9

CONTINUOUS INTEGRATION—RUNNING NTIME IN MS BUILD


Assume that you have saved this build file as:

“NTimeBuild.build” at the location <<BUILD_FILE_FOLDER>>

3. Open the “Visual Studio Command Prompt” tool.4. Type the following command in that tool:

MSBuild.exe “<<BUILD_FILE_FOLDER>>\NTimeBuild.build” /t:NTimeTarget5. Execute the above command by pressing the

“Enter” key. All the assemblies mentioned in the build file will be performance tested and results will be displayed as shown in Figure 4.

10

Figure 4: Snapshot of NTime Result in MS Build

11

NTime allows the development team to get a preview of what the performance of a particular scenario would look like. Early diagnosis of a performance bottleneck will reduce effort in later stages when only profiling could identify the bottlenecks. It also enables an architect to performance budget a scenario and give the developer meaningful goals to achieve, since he/she now has an option to verify that the scenario is functioning well within the performance budget. Integrating NTime into a continuous delivery framework will ensure that the frequent changes to the code will not create surprises later in development or during the release of the product. NTime results could also be part of the exit criteria for development. Doing so will provide the reviewer with vital statistics of the code performance (or lack thereof).

NTime, as noted before, is neither a performance benchmarking profiler nor a full-fledged performance testing suite. It does not replace end-to-end performance testing tools like Load Runner. Instead, NTime, as a tool, can be used to generate performance metrics at individual and unit-level source code. It helps the developers to determine whether the source code is ready for performance test.

References 1. Section 10.12. Creating Performance Bottleneck tests with NTime- Windows Developer Power Tools. 2. Code Project article, NTime- Performance unit testing tool, written by Adam Slosarski.

BENEFITS OF NTIME

CONCLUSION


About Sapient Global MarketsSapient Global Markets, a division of Sapient® (NASDAQ: SAPE), is a leading provider of services to today’s evolving financial and commodity markets. We provide a full range of capabilities to help our clients grow and enhance their businesses, create robust and transparent infrastructure, manage operating costs, and foster innovation throughout their organizations. We offer services across Advisory, Analytics, Technology, and Process, as well as unique methodologies in program management, technology development, and process outsourcing. Sapient Global Markets operates in key financial and commodity centers worldwide, including Boston, Chicago, Houston, New York, Calgary, Toronto, London, Amsterdam, Düsseldorf, Geneva, Munich, Zurich, and Singapore, as well as in large technology development and operations outsourcing centers in Bangalore, Delhi, and Noida, India.

For more information, visit sapientglobalmarkets.com.

© 2011 Sapient Corporation. Trademark Information: Sapient and the Sapient logo are trademarks or registered trademarks of Sapient Corporation or its subsidiaries in the U.S. and other countries. All other trade names are trademarks or registered trademarks of their respective holders.

GLOBAL MARKETS 1

GLOBAL MARKETS 1

DüsseldorfHammer Straße 1940219 DüsseldorfGermanyTel: +49 (0) 211 540 34 0

GenevaSapient Switzerland S.a.r.l.Succursale Genèvec/o Florence Thiébaud, avocaterue du Cendrier 151201 GenevaSwitzerlandTel: +41 (0) 58 206 06 00

HoustonHeritage Plaza1111 Bagby Street Suite 1950Houston, TX 77002Tel: +1 (713) 493 6880

LondonEden House8 Spital SquareLondon, E1 6DUUnited KingdomTel: + 44 (0) 207 786 4500

Los Angeles1601 Cloverfield Blvd.Suite 400 SouthSanta Monica, CA 90404Tel: +1 (310) 264 6900

Munich Kellerstraße 2781667 MünchenGermanyTel: +49 (0) 89 552 987 0

New York40 Fulton Street22nd FloorNew York, NYTel: +1 (212) 560 5700

SingaporeAir View Building #02-012 Peck Seah StreetSingapore 079305Republic of SingaporeTel: +65 (3) 104 5308

Toronto129 Spadina AvenueSuite 500Toronto, Ontario M5V 2L3CanadaTel: +1 (416) 645 1500

ZürichSapient Switzerland GmbHSeefeldstrasse 358008 ZurichSwitzerlandTel: +41 (58) 206 06 00

Global Offices

AmsterdamSapient Netherlands B.V.Damrak 701012 LM AmsterdamTel: +31 (0) 20 330 0011

BangaloreSalarpuria GR Tech Park6th Floor, “VAYU” Block#137, Bengaluru 560066KarnatakaIndiaTel: +91 (080) 410 47 000

Boston131 Dartmouth Street3rd FloorBoston, MA 02116Tel: +1 (617) 621 0200

Calgary888 3rd Street SWSuite 1000Calgary, Alberta T2P 5C5CanadaTel: +1 (403) 444 5574

Chicago30 West Monroe,12th floorChicago, IL 60603Tel: +1 (312) 458 1800

DelhiTowers D & E,DLF Cyber GreensDLF Phase IIISector 25-AGurgaon 122002HaryanaIndiaTel: +91 (124) 416 7000

unit testing for performance using ntime - · pdf fileunit testing for performance using ntime...

Documents