Industry

Retail

Solution & Services Performance/Load Testing

Software Test Management

Test tool evaluation

Tools

Microsoft Visual Studio Ultimate version

CASE STUDY

Leading Retailer Chooses

Microsoft Visual Studio for Load Testing Critical Web Services

Background

RTTS’ client operated numerous heterogeneous customer-

facing (i.e. revenue producing) web applications, as well as

administrative and customer support applications, that relied

on a common set of services, including customer account

retrieval, delivery date availability, and address validation.

Using IBM’s Integration Bus solution (formerly known as

WebSphere Message Broker), RTTS’ client was launching an

enterprise server bus (ESB), so that these individual

applications (and future applications) could more easily scale

in terms of business growth.

Challenges

High availability and responsiveness were critical factors to the customer’s business model, since

these factors directly affected order conversion for the customer-facing applications, as well as mean

time to resolution regarding customer service inquiries. RTTS’s customer required a load testing

solution that possessed a minimal total cost of ownership (TCO) that could scale in proportion to its

business growth and the consumption of the web services provided by the ESB.

Licensing expenditures and load generation infrastructure costs were paramount in being able to

meet budget constraints, while the ability to scale to the transaction rates and concurrency were

imperative to the application and infrastructure teams. The short-term load testing goal was to be

able to support the demands of at least one (1) of the customer’s primary brand’s e-retail web site

using a single instance of the IBM Integration Bus. Peak usage had been observed to be

approximately 7,000 online customers.

Strategy

RTTS put forth a proposal that would assess the available load testing solutions. Proof-of-concepts

were arranged that addressed the gamut of open source load testing solutions and commercial load

testing solutions. The results of the proof-of-concepts were presented to the stakeholders in

conjunction with the projected costs of the solution in order to determine the best return on

investment. RTTS also took ownership of the rollout of the load testing solution by providing

professional services that installed the testing products and engineered the baseline performance

tests.

Solution

Several load testing solutions were evaluated using the criteria set forth by RTTS’ customer. Open

source, commercial, and cloud-based load testing solutions were evaluated via proof-of-concepts

against a subset of the ESB’s suite of web services. Load test solutions that were provided by open

source testing tools were deemed a risk in terms of having product support. Cloud-based testing

solutions turned out not to be a fit a the time due to concerns regarding security, proprietary scripting

languages, and the client’s current corporate culture. However, RTTS’ client did express interest in

having the flexibility to move to a cloud-based solution, especially regarding its load generation

infrastructure. In the end Microsoft Visual Studio Ultimate was selected to be the best fit based upon

its capabilities, familiarity, and cost.

The initial cost of Microsoft Visual Studio Ultimate was significantly less expensive when compared

to some of the more established load testing solutions in the market. Microsoft’s licensing model

permitted unlimited numbers of virtual users, so that testing could easily scale along with the growth

of the customer’s business. Along with the ability to systematically test the ESB using a familiar

Visual Studio IDE, the ability to add capacity to its load generation infrastructure was seamless.

RTTS was able to incrementally add to the customer’s load generation infrastructure by simply

spinning up additional virtual machines and adding them to the current Visual Studio Load Test Rig.

If and when the need arises, these tests could be ported to the Windows Azure cloud platform for

greater flexibility for implementing its load generation infrastructure.

Benefits

RTTS successfully installed and configured the necessary Microsoft Visual Studio components and

was able to jumpstart the load testing initiative by testing a subset of the ESB web services related

to product availability, delivery dates, and shipping options for one (1) of the customer’s family of

brands. Using a Visual Studio Load Test Rig of six (6) agents, load tests reaching 12,000 concurrent

users were executed. Microsoft Visual Studio’s load testing capabilities were able to assist in

discovering bottlenecks related to the threading model being used by the ESB’s database

connection pool, as well as within the delivered HTTP configuration of the ESB’s web interface.

Once the environment was optimized, 90% of all web service requests were returned in under one

(1) second. The IBM Integration Bus was shown to support the processing of 55 web service

requests per second with its CPU utilization averaging about 60%.

At the conclusion o the project RTTS was tasked with leading the testing effort by augmenting the

collection of web services being tested, incorporating additional retail brands and back-office

applications, and subsequently increasing the end-user consumption of the web services.

RTTS is a Microsoft Gold Partner. Since 2006, RTTS has been an education & consulting partner with

Microsoft – implementing solutions and providing educational services around Microsoft's Visual Studio

testing tools. RTTS has worked with the Visual Studio team to develop top-notch solutions around their

excellent toolset. For more information about our solutions and services, go to www.rttsweb.com

copyright Real-Time Technology Solutions, Inc. 2014