service virtualization - the economics of software testing

COPYRIGHT 2013, WINTERGREEN RESEARCH, INC. 1 www.wintergreenresearch.com www.wintergreenresearch.com/blog tel 781-863-5078 781 863 1235 email: [email protected] Lexington, Massachusetts

White Paper

By: WinterGreen Research, Inc.

The Economics of Software Testing:

Service Virtualization

Return on Investment Analysis


Executive Summary: The Driving Forces For Service Virtualization

Software testing and service virtualization have become core driving forces to improving the efficiency

of business operations. Software testing comes to the forefront as a way to improve automated

process implementation because services virtualization can be used directly to grow the business by

improving project quality. Service virtualization is used to catch programming errors early in the test

cycle, prevent shipping software that has errors in it, support mobile application development, respond

to complexity, reduce cost, mitigate risk, and improve time to market. Service virtualization is a vital

part of the business automation process, freeing people for other

tasks. As part of automated process, businesses need to achieve

better, faster testing of software, thereby reducing time to market

and freeing development and testing people resources. Service

virtualization is good because with it, it takes less time and fewer

people to do project software testing. Since software testing is

done faster and better, service virtualization frees people resource

for other project tasks.

Complexity The increasing complexity of IT systems is a fact of life. Addressing

complexity to hide it and thereby contribute to software development

efficiency is addressed with component architectures including SOA. Yet,

for an organization to realize the full potential of SOA depends on the

ability to test these composite applications early in the development

lifecycle, before all of the services/components are even available.

Cost

Testing applications that are based on composite architectures can require

a large number of assets and resources, which in today’s economy are just

not feasible as the business continues to be faced with pressures to do

more with less. Constraints in budgets mean that management of

software testing costs and quality is a necessity. The ability to virtualize applications, sometimes even before

they are ready is more critical than ever, because it has the dual effect of improving quality while lowering cost.

Risk

The quantity of new code and existing code changes are continually an issue. As applications become more

complex, software testing is a risk management concern, contributing to the overall success of the business by

meeting customer demands while adhering to, security, and regulatory compliance requirements.

Service Virtualization Executive Summary: Provides The Equivalent Of Several New People Hired Deals With Complexity Using Automated Testing Reduces IT Cost Manages Risk in the Business Improves Time to Market


Time To Market

Businesses have an ongoing need to grow the business and to provide flexible response to changing

market conditions. Speeding new solutions to market is a priority. This creates software development

challenges with the move to shorten product delivery cycles. Pressures on software testers result from

the need to leverage the Internet, implement mobile apps, and manage a large and growing number of

customers. The testing process cannot wait for completed systems, we need to test early and with

increased automation.

Software Testing and Service Virtualization Research Approach /

ROI Calculator Software testing savings of 30% can be achieved by implementing a services

virtualization solution. This white paper analysis is backed by an online live ROI

calculator that identifies the metrics that have the biggest impact on costs and

savings for software testing and shows how savings accrue under various scenarios.

Assumptions about service virtualization include # Projects Per Year, Average

Project Cycle, Number of Q/A technicians per Project, and the time spent on testing

with and without service virtualization automation in the testing process. In the

WinterGreen Research testing software calculator there are 48 metrics that can be

locked down or adjusted as variables to determine a probable ROI for the

automation scenarios considered for various situations. The resultant ROI is not a

black box, it is a set of calculations that can be examined and customized to match

your particular situation. It can be used by to get the basis of potential savings

accrued by changing 3 or 4 variables to customize the calculations, or it can be used

thoughtfully and carefully changing as many variables as desired.

Service Virtualization Driving Forces

The driving forces for service virtualization are core to running the business: using computers and

software to grow the business, catching programming errors early in the test cycle, and supporting

mobile application development. Service virtualization supports responding to complexity. It is used for

reducing cost, mitigating risk, and improving time to market as a way to grow the business.

Need To Do More With The Same Amount Of Resources

ROI

Calculator

Savings of 30%

Gist of Savings or

In Depth

Calculations


Leveraging Service Virtualization for

Application Testing

Addressing the challenges of software development means that

software needs to run with fewer bugs and to do so, bugs need

to be found earlier in the development / testing cycle. Installed

capacity needs to be tested rapidly and brought up and running

in a timely manner. The need to test everywhere that new code

touches existing code is daunting; the need to test new code

with a range of different

types of applications and

API modules can be time

consuming. Existing

application components

may not be readily

available for

interoperability testing.

When they are available it

may be weeks later than

when the developer is

ready - significantly delaying the move of newly developed business

capabilities or entire applications into production.

The elements of application testing that can be managed efficiently

using software virtualization solve all the

issues listed above: software virtualization

supports testing faster, finding errors faster,

fixing errors faster, and managing testing

across different platforms. Managers

contending with slow revenue growth need

efficient ongoing operations so they can

implement projects that have hope of growing the business. There is a continuing

need to do more with the same amount of resources and flat or decreasing

budgets. Creating better testing efficiency can expand the amount of available

resources, sometimes exponentially. Freeing testers to test may mean getting to market sooner or

increasing the testing coverage to ensure higher quality releases.

CIOs are wrestling with how best to function in an environment where all new hiring is frozen. This lack

of new labor resource stymies innovation and stymies any but the most compelling upgrades of the data

center. To free up testers provides at least a cadre of technical resource possibly able to handle the

Service Virtualization Driving Forces: Rising costs Quality issues Complex applications Modular / composite applications (including SOA) Fast development iterations Rapid response to changing

Addressing

challenges

Testing rapidly

Growing the Business Is a

Huge Priority

Application testing is More

Efficient if It Catches

Programming Errors Early in

the Test Cycle

Mobile Application

Development Brings Special

Complexities To Application

Testing.


processing and service demands not now being met, while helping IT to continue to support the

enterprise to remain a viable and competitive business entity. Service virtualization has emerged as a

way to expand existing technician resources and mange more effectively. Services virtualization

supports simulation of data and application integration diminishing volume impact on software testing,

supporting integration of the business and IT.

Application testing bottlenecks are becoming a key focus area for many managers. The interest is in

improving the efficiency of project management in any way possible. Through service virtualization,

testers can simulate the data and application integration points making it possible to produce more

testing volume, improving testing efficiency.

Software developers and software testers have limited access to some code. A key pain point is to

achieve access to code that is not yet available. That part of a packaged application that the tester does

not have non-production access to is a problem. The cost of leveraging pay per use services for testing

is a problem. Services virtualization helps address the issues of code availability. Thus, service

virtualization is much appreciated when the tester is under the gun to produce high quality software

quickly. Software virtualization overcomes testing problems and software delay problems not

previously possible to overcome. Services virtualization does support the implementation of new code

and the adjustment to existing code in a far more efficient manner than has previously been possible.

Service virtualization provides a way to manage available options and

quantify the value of speeding the software testing process to

accelerate the delivery of new software applications into production.

Testing Solutions

Rational automates business processes with systems through

application development, testing, and production delivery. This is a

complex task. The aim is to pull together servers and applications

from various production groups. Moving testing earlier in the process

means bugs are easier to fix, errors have less impact on the software

architecture enabling developers to perform tasks more effectively and efficiently.

Do more with the same

amount of resources

Further integrate Business

and IT

Discovering Bugs Early in the Process Improves Testing

Effectiveness


IBM Rational Is Automating Test Environments with Service

Virtualization

IBM Rational Test Workbench is automating test environments

through service virtualization. Service virtualization is being used to

improve the economics of software testing, freeing people. In an

environment with a hiring freeze, freeing up testers is able to support

more resource for projects. People can be retrained using new just in

time training techniques, and those people utilized in ways that may

not have been thought possible before.

Service virtualization is efficient. It is effective. It saves customers

millions of dollars per year. It is easy to use. It frees

people resources to be available to work on projects that

may grow the business.

IBM Rational service virtualization capabilities powered by Green Hat replace the need to manually

implement, configure and administer test environments

which can be extremely time consuming, may require

specialized skills, and are very expensive. The elimination of

manual process enables mission critical application testing to

be automated and managed in an agile manner. Multiple

iterations, each having different conditions to be validated

are supported easily with the IBM solutions.

Testing From The User’s Perspective

Developers and testers who are testing from the user

perspective are able to find and fix problems much earlier in

the development and test cycle, creating a lower cost

development. Every error testers find improves the brand.

User interface testing is part of the testing challenge that is

sped up by leveraging service virtualization. By having a

seamless and automated developer and tester tool

environment, bug identification and fixes become more

efficient. The development cycle benefits from test

automation that is able to keep pace with agile and iterative

cycles.

Agile Development Demands Complex Software Testing

IBM Rational Test

Workbench Tools Are

Same Tools Developers

Are Using

No one ever really

thought it possible to

recreate all the different

server and application

configurations virtually

Now IBM has made this

possible with the Green

Hat acquisition and

introduction of Rational

Test Workbench


The Benefits of IBM Rational Service Virtualization:

The cost savings are achieved by automating the process of recreating every existing configuration of

the production environment for complete software testing. Using service virtualization saved over $6

million in three years according to the above example. Until recently, no one ever really thought it

possible to recreate all the different server and application configurations virtually.

This means that it is possible to simulate a real production environment as soon as the developer is

ready to test their code even if dependencies, services, components and data structures are not ready

or available. The developer is able to catch errors and fix them earlier, even before the formal testing

cycle begins.

Table 2

IBM Rational Service Virtualization Application Project

Testing

Full remote troubleshooting capabilities Network loopback Test access Test signal generation Testing integrated into the platform Ensure new application meets all operational requirements via rigorous testing Source: WinterGreen Research, Inc.

COPYRIGHT 2013, WINTERGREEN RESEARCH, INC. 8 www.wintergreenresearch.com www.wintergreenresearch.com/blog tel 781-863-5078 781 863 1235 email: [email protected] Lexington, Massachusetts RAL14071-USEN-01

Dramatic Savings Brought by IBM Rational

Automated Test Solution

The ROI analysis above demonstrates dramatic savings from use of

IBM Rational Test Workbench. Quality of software code can be

dramatically improved by pushing defect detection close to the point

of introduction. Service virtualization directly relates to increasing

collaboration between development and test teams. Discovering

major issues needs to occur early in the code development process.

Defects found later in the process take more effort, more time and

cost more to fix. Positive ROI is achieved through the ability to

minimize the cost and time to identify and fix errors, and diminish

subsequent impacts of the defect.

Use of integrated simulation solutions on the cloud provides

significantly more attractive ROI. Online sliders are included as part

of the web site ROI that accompany this paper. WinterGreen

Research endorses the IBM Service Virtualization offering as the best

testing approach, particularly for software testing organizations using

their own physical infrastructure to create unique test environments

for each new application. It is as critical when leveraging third party applications, especially when a

system integrates with a partner, to have service virtualization available. Because, even though a tester

may have access to the internal corporate piece of the software solution, that tester may not have

access to code so that testing against the third party piece can proceed apace. This is a very common

and important example. Testing approaches using manual process have become particularly onerous as

agile development is implemented, leading to the need for testing modernization.

In conclusion, automation of testing using service virtualization software yields

significant cost savings benefits. It keeps the software development process balanced.

The detection of errors earlier in the testing process results in better overall systems

deployment, faster software utilization, and overcomes testing bottlenecks.

Simple Automation Of

Testing Software Can

Yield Significant Benefits

In Keeping The Process

Balanced To Detect

Errors Earlier In The

Testing Process,

Resulting In Better

Overall Utilization of

Newly Developed Or

Changed Software

WinterGreen

Research

endorses

IBM Service

Virtualization

IBM Rational Automated Test Solution, Powered by Green

Hat Technology Reduces The Testing By 30% And The Q/A

New Functionality Testing Is Reduced by 25%. Virtualizations

Savings Average 30%, Range from 20% to 500%

service virtualization - the economics of software testing

Documents