offshore remote application testing

8/12/2019 Offshore Remote Application Testing

http://slidepdf.com/reader/full/offshore-remote-application-testing 1/12

A M R R e s e a r c h R e p o r tA p r i l 2 0 0 5

by Lance Travis

Remote application testing provides companies witha valuable, lower risk way to establish a relation-ship with an offshore outsourcing service provider.Benets include lower testing costs and improvedquality. More important, testing services provide a way for companies to mature their offshore outsourc

ing skills in preparation for a long-term developmenrelationship.

Offshore Remote Application Testing Can Cut Cost ofQuality by Up to 75%



© Copyright 2005 by AMR Research, Inc.

No portion of this report may be reproduced in whole or in part without the prior written permission of AMR Research. Any writtenmaterials are protected by United States copyright laws and international treaty provisions.

AMR Research offers no specific guarantee regarding the accuracy or completeness of the information presented, but the professional staffof AMR Research makes every reasonable effort to present the most reliable information available to it and to meet or exceed anyapplicable industry standards.

AMR Research is not a registered investment advisor, and it is not the intent of this document to recommend specific companies forinvestment, acquisition, or other financial considerations.



TheBottomLine

AMR Research Report | April 2005 1© 2005 AMR Research, Inc.

Offshore Remote Application Testing CanCut Cost of Quality by Up to 75%by Lance Travis

Offshore software testing saves companies up to 75% over in-house

costs, improves quality, and is a lower risk way to develop mature offshore

outsourcing skills.

Software defects continue to proliferate, but the cost of ensuring quality is high. Software testing can account for 30% of develop-ment time and many companies’ staff testing positions withhigh-cost, on-site independent contractors. Companies must

decide if offshore testing provides a solution to their cost of quality problems.

After interviewing 14 service providers that offer offshore testing services and reviewingthe results of 16 customer engagements, AMR Research concludes the following:

• Low cost, faster test cycles, and improved quality are benefits from offshore testing.

• Segment testing reduces the risk of offshore outsourcing.

• Engagement size and add-on services distinguish the service providers.

Software quality issues continue to plague CIOs andsoftware product development organizations

Academics report that in 2002, software defects cost $60B to x, and 50% of softwaredevelopment costs were related to nding and xing defects. Obviously, many com-panies aren’t getting the job done with in-house resources or on-site contractors. eymust decide whether the benets of using offshore software testing services outweigh

the risks.

Low cost, faster test cycles, and improved quality arebenefits from offshore testing

Companies interviewed for this Report consistently reported cost savings, faster testcycles, and improved quality as the benets they received from offshore testing.

ExecutiveSummary



AMR Research Report | April 20052 © 2005 AMR Research, Inc.

Many companies report 60% to 75% savings by moving testing offshore

More than half of the companies that we talked to realized savings between 60% and75% over in-house costs. Rather than letting people go, one manufacturing company was able to move the replaced in-house staff to other projects and stil l save $300K

per year. Some companies report that rather than realizing savings they maintainedspending at roughly the same levels, they increased the level of testing because of thelower costs associated with offshore resources.

Savings depend on the percentage of offshore resources and on the amount of overheadassociated with managing the offshore relationship. Some projects, such as quality con-sulting projects, require a much higher on-site presence and result in smaller savings.

Leveraging staffing exibility is another way companies saved money using offshoretesting providers. Rather than keeping underutilized in-house resources, companiesramped up and down outsourced resources, depending on the recurring demands of

their development cycle.

Faster test cycle times result from offshore testing

Companies are able to leverage time zone differences to accelerate test cycles in a follow-the-sun philosophy. Daytime testing occurs in India and is then handed over to the United Statesat the end of the day in order to leverage the 9.5- to 12.5-hour time difference between thetwo countries. One company reduced its test cycle by half, and then improved quality byadding more test scenarios.

When leveraging the 24-hour clock, companies had to make arrangements for off-hours support of their test environment. Before it made arrangements, one companylost 48 hours’ worth of testing because a server failed on a Sunday evening (U.S.time) of a three-day weekend. e Indian test team was unable to work Monday andTuesday in India because no one was available to x the problem in the United Statesuntil Tuesday morning U.S. time (Tuesday night India time).

All of the service providers interviewed for this Report offer services to automate tests. After using the offshore service provider to automate its 3,500 tests, one companyreduced its test cycle from 90 days to 29. e cost of automating the tests was muchless than the cost of continually running them manually.




Quality improvement results from additional testing, and eventuallyroot-cause analysis

Companies reported immediate quality improvements as a result of test automationand reduced cycle times. Test automation allows them to run more tests during a test

cycle. Because they are able to run more tests, they are able to nd more problems.One manufacturing company reported that because it never allocated enough resourcesto in-house testing, it never would have been able to automate and test as thoroughlyas the offshore provider. A nancial services company reported a dramatic decrease inthe number of post-go-live reported defects after the offshore provider automated thecompany’s test scripts.

As the offshore testers become more familiar with the applications being tested, theybegin to help their customers nd the root cause of defects and eliminate future prob-lems. However, this ability doesn’t happen right away. Companies reported that ittook between 14 months and 3 years before the offshore testers had sufficient familiar-ity to be effective in nding the root cause of problems.

Segment testing reduces the risk of offshoreoutsourcing

Testing work can be outsourced offshore with less risk than application development ormaintenance because of the following:

• The work can be easily segmented into process consulting, test execution, test

automation, test development, and quality improvement consulting. Processconsulting allows companies to leverage the service provider’s best practices toimprove how they test. Process consulting, test execution, and test automationrequire minimal interaction with business users, allowing an IT organization toestablish a relationship with an offshore provider without involving the business users.Test development requires working with the business user to understand the purposeof the software to be tested and their acceptance criteria to be understood.

• Testing doesn’t require the offshore resources to touch the application sourcecode. The offshore resources can gain experience with the customer’s developmentprocess and its applications without running the risk of breaking or corrupting anapplication.

• Most companies underinvest in testing. An effective offshore testing program canconvince outsourcing naysayers that offshore work can be effective.

• Testing engagements can slowly ramp up. It is possible to start small and workup to a larger engagement. The number of resources supplied by the outsourcer forthe customer engagements studied in for this Report ranged from 6 to 350 people.It is also possible to start with lower value activities such as test execution, and rampup to higher value activities such as root-cause analysis (see Figure 1).




Not all companies that we talked with were convinced that testing is a good candidatefor a company’s rst foray into offshore outsourcing. One company had difficultysending test execution offshore because its development process and documentation were nonexistent. It was only after the offshore company was able to drive processimprovements into the development organization that the company was able to out-

source test execution.

Savings fromProblem

Elimination

Savings fromProcess

Automation

Savings fromLabor

Arbitrage

Savings fromProcess

Improvement

QualityProcess

Consulting

TestExecution

TestAutomation

TestDevelop-

ment

Root CauseAnalysis and

QualityImprovements

Q u a

l i t y I m p r o v e m e n

t s

Maturing Relationship

Outsourced Task

Figure 1: Value increases as relationship matures

Source: AMR Research, 2005




Separate testing and development organizations importantto customers

Many customers selected their testing service provider because it offered a testing serviceseparate (reporting to different management) from its development organization. Two of

the companies selected one offshore company for development and another for testing.e customers felt that having separate organizations ensured that testing and quality

concerns would not get subordinated to development schedule concerns. Having mul-tiple companies involved created a healthy competition that beneted the customer.

One company using separate organizations for testing and development felt that it wasessential to use in-house resources as the quality advocate. e person in charge ofthe project had trouble getting the Indian resources to push hard enough for processimprovements.

Despite the lower risk, strong on-site leadership is still required

All but one company interviewed stated that having the outsourcing company sup-ply on-site resources was essential. e on-site resources are necessary to manage theow of work to and from the offshore locations. In addition to having on-site person-nel, companies recommend having strong in-house managers handle the relationship.Companies must establish ground rules for the relationship and cannot just throwthings over the wall to the outsourcer and expect success. For test execution projects,companies typically are able to move 70% to 80% of the test resources offshore, butnd that keeping the remaining 20% to 30% in-house is essential.

Offshore testing does require some infrastructure investment

e offshore resources need access to different systems than offshore developers require.For example, one nancial services company provided access to the systems that areintegrated with the to-be-tested systems to ensure a complete systems test. In this case,the company extended its private network to India.

Some companies had problems with sending test data to India. Either governmentregulations or contracts with their customers prevented them from moving test dataoffshore. In cases such as these, the offshore providers offered tools to facilitate the cre-

ation of unrestricted test data based on live data.




Engagement size and add-on services distinguish theservice providers

AMR Research interviewed a broad cross-section of service providers for this Report: Accenture , AppLabs, Cognizant , HCL Technologies , Headstrong , Infosys, ITCInfotech , Keane, Patni , Rapidigm , Sapient , Satyam, Tata Consultancy Services (TCS), and Wipro . All the service providers interviewed offer a suite of test consult-ing and software lifecycle testing services (e.g., manual testing, functional testing,performance testing, acceptance testing, and test automation). All have provided testservices for multiple clients, with AppsLab, Cognizant, Infosys, Keane, Patni, Rapidigm,Satyam, TCS, and Wipro having more than 50 customers.

e service providers support multiple engagement models: time and materials, xed-priced projects, and dedicated test centers. However, a majority of the offshore testingengagements are time and materials contracts.

e service providers examined fall into three categories: large offshore rms, smalleroffshore rms, and traditional consulting rms.

The larger rms support the largest engagements

Cognizant, Infosys, TCS, and Wipro are primarily focused on large engagements.eir average engagements number over 50 test resources and last a year or more, and

their largest engagements are double to triple the size of the smaller rms’. Considerthese rms for long-term, large test engagements (see Table 1). You can start small with these rms, but unless you show solid plans to scale up the relationship, you maynd that your requirements are mismatched with their delivery capabilities.

Firms supporting smaller engagements are ne for short-term staffaugmentation and midsize, longer term engagements

e average deals of smaller rms—AppLabs, HCL Technologies, Headstrong, ITCInfotech, Patni, Rapidigm, and Satyam—involve 5 to 10 test resources and last 6to 9 months. eir largest deals range from a low of 15 resources (Headstrong andRapidigm) to over 60 (Patni and Satyam). Consider these companies for small staffaugmentation projects. Consider the high end of this category (AppLabs, HCL, Patni,and Satyam) for larger, longer term projects.

Note that project size does not always determine project complexity. Smaller projectscan be quite complex, involving multiple applications, geographies, and quality con-sulting. Despite the relatively smaller size of their engagements, Patni and Satyamhave some of the more complex engagements.




Traditional consulting rms enter the offshore testing market

Accenture, Keane, and Sapient fall into this category. ey have recently supple-mented their implementation services with stand-alone offshore testing capabilities. Accenture created a dedicated test center of excellence in India in 2004 and now has

over 500 test resources. Keane also created a dedicated test center in India in 2004and has more than 300 test resources. Sapient provides testing services through itsdevelopment resources. To date, its testing engagements have been small compared tothe large offshore rms. Accenture's and Keane’s largest engagements involve 30 and26 test resources, respectively. Use a testing engagement with these providers primarilyas an entry point into their application implementation and consulting services.

In addition to the standard test consulting and quality offerings, some of the providersoffer unique test services that may be of specic interest:

• Usability testing —Cognizant, Patni (in 2005), Rapidigm, Satyam

• Compliance testing —Accenture, Cognizant (Health Insurance Portability and Accountability Act [HIPAA], CFR 21), Infosys (HIPAA, CFR 21, Sarbanes-Oxley Act[SOX]), Patni (HIPAA), TCS (SOX), Satyam (CFR 21), and Wipro (SOX in 2005)

• Packaged application testing —All of the service providers offer some level of pack-

aged application testing, but those planningSAP and Oracle upgrade testing shouldconsider the service providers with the deepest SAP and Oracle implementationskills: Accenture, Cognizant, HCL, Infosys, Patni, Rapidigm, Satyam, TCS, and Wipro. Note that Headstrong only offersSiebel testing services.

• Business strategy and test strategy alignment offered by Accenture, Infosys,Keane, Satyam, TCS, and Wipro —In addition to consulting about test technologyand test processes, they can help you prioritize your test strategy based on the impor-tance of specif ic applications and application functionality to your business strategy.

Source: AMR Research, 2005

Table 1: Attributes that indicate service maturity

CompanyYear service

started

Number of testcenter employ-

ees

Number of re-sources in larg-

est deal

AppLabs 2000 450 75Cognizant 2002 1200 122

HCL Technologies 2001 250 50

Headstrong 2001 50 15

Infosys 2001 1200 135

ITC Infotech 2003 117 16

Patni 2002 685 65

Rapidigm 2002 45 15

Satyam 2001 650 70

TCS 2000 1350 120

Wipro 1997 2460 350




Offshore testing services have lower-than-expected attrition rates

e service providers have focused on making testing an attractive career path andminimized attrition within their organizations. e companies offer extensive techni-cal and business training and career advancement opportunities in order to attract and

retain quality resources. As a result, attrition is often lower in the test organizationsthan in the rest of the Indian companies. Infosys, ITC Infotech, Rapidigm, Satyam,and TCS reported the lowest attrition rates within their testing organizations. Long-term customers benet because of consistency of staff on their projects.

Common tools facilitate workow between on-shore and offshore

All of the service providers interviewed support standard third-party test tools fromcompanies such asCompuware , Empirix , IBM Rational , Mercury , and Segue, andopen source tools such as JUnit . e providers use standard tools such as Mercury

TestDirector, IBM Rational’s ClearQuest, and Excel spreadsheets sent via e-mail to facili-tate the ow of information between locations. Whenever possible, companies shouldleverage the relationships the service providers have with tool vendors when establishingthe tools to be used in the engagement.

Closing Comments

Testing is a low-risk, high-reward entry into offshore services. Companies that havebeen struggling to develop an offshore outsourcing strategy should look to a limitedtest engagement as their rst step. Outsourcing test execution and test automationlimit dependence on the business units for support, offer signicant cost savings andimproved quality, and give companies a chance to hone outsourcing skills before mov-ing on to more extensive offshore engagements.



Company List Accenture www.accenture.com

AppLabs www.applabs.com

Cognizant www.cognizant.com

Compuware www.compuware.com

Empirix www.empirix.com

HCL Technologies www.hcltechnologies.com

Headstrong www.headstrong.com

IBM Rational www-306.ibm.com/software/rational

Infosys www.infosys.com

ITC Infotech www.itcinfotech.com

Keane www.keane.com

Mercury www.mercury.com

Oracle www.oracle.com

Patni www.patni.com

Rapidigm www.rapidigm.com

SAP www.sap.com

Sapient www.sapient.com

Satyam www.satyam.com

Segue www.segue.com

Siebel www.siebel.com

Tata Consultancy Services (TCS) www.tcs.com

Wipro www.wipro.com



AMR Research provides world-class research

and actionable advice for executives tasked

with delivering enhanced business process

performance and cost savings with the aid

of technology. 5,000 leaders in the Global

1000 put their trust in AMR Research’s integ-

rity, depth of industry expertise, and pas-

sion for customer service to support their

most critical business initiatives, including

supply chain transformation, new productintroduction, customer profitability, com-

pliance and governance, and IT benefits

realization. More information is available at

www.amrresearch.com.

Your comments are welcome. Reprints

are available. Send any comments or

questions to:

AMR Research, Inc.

125 Summer Street

Boston, MA 02110 Tel: +1-617-542-6600

Fax: +1-617-542-5670

www.amrresearch.com

555 Montgomery Street

Suite 650

San Francisco, CA 94111

Tel: +1-415-217-3737

Whittaker HouseWhittaker Avenue

Richmond, Surrey TW9 1EH

United Kingdom

Tel: +44 (0) 20 8822 6780

Fax: +44 (0) 20 8822 6790

AMR-R-18109

Acronyms and Abbreviations

HIPAA Health Insurance Portability and Accountability Act

SOX Sarbanes-Oxley Act

offshore remote application testing

Documents