software test in practice

7/29/2019 Software Test in Practice

1/18

Hindawi Publishing CorporationAdvances in Software EngineeringVolume 2010, Article ID 620836, 18 pagesdoi:10.1155/2010/620836

Research ArticleSoftware Test Automation in Practice: Empirical Observations

Jussi Kasurinen, Ossi Taipale, and Kari Smolander

Department of Information Technology, Laboratory of Software Engineering, Lappeenranta University of Technology,P.O. Box 20, 53851 Lappeenranta, Finland

Correspondence should be addressed to Jussi Kasurinen, [email protected]

Received 10 June 2009; Revised 28 August 2009; Accepted 5 November 2009

Academic Editor: Phillip Laplante

Copyright 2010 Jussi Kasurinen et al. This is an open access article distributed under the Creative Commons AttributionLicense, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properlycited.

The objective of this industry study is to shed light on the current situation and improvement needs in software test automation. Tothis end, 55 industry specialists from 31 organizational units were interviewed. In parallel with the survey, a qualitative study wasconducted in 12 selected software development organizations. The results indicated that the software testing processes usuallyfollow systematic methods to a large degree, and have only little immediate or critical requirements for resources. Based onthe results, the testing processes have approximately three fourths of the resources they need, and have access to a limited, butusually sufficient, group of testing tools. As for the test automation, the situation is not as straightforward: based on our study, theapplicability of test automation is still limited and its adaptation to testing contains practical difficulties in usability. In this study,we analyze and discuss these limitations and difficulties.

1. Introduction

Testing is perhaps the most expensive task of a softwareproject. In one estimate, the testing phase took over 50%of the project resources [1]. Besides causing immediatecosts, testing is also importantly related to costs related topoor quality, as malfunctioning programs and errors causelarge additional expenses to software producers [1, 2]. Inone estimate [2], software producers in United States loseannually 21.2 billion dollars because of inadequate testing

and errors found by their customers. By adding the expensescaused by errors to software users, the estimate rises to59.5 billion dollars, of which 22.2 billion could be saved bymaking investments on testing infrastructure [2]. Thereforeimproving the quality of software and effectiveness of thetesting process can be seen as an effective way to reducesoftware costs in the long run, both for software developersand users.

One solution for improving the effectiveness of softwaretesting has been applying automation to parts of the testingwork. In this approach, testers can focus on critical softwarefeatures or more complex cases, leaving repetitive tasks tothe test automation system. This way it may be possible to

use human resources more efficiently, which consequentlymay contribute to more comprehensive testing or savings inthe testing process and overall development budget [3]. Aspersonnel costs and time limitations are significant restraintsof the testing processes [4, 5], it also seems like a soundinvestment to develop test automation to get larger coveragewith same or even smaller number of testing personnel.Based on market estimates, software companies worldwideinvested 931 million dollars in automated software testingtools in 1999, with an estimate of at least 2.6 billion dollars

in 2004 [6]. Based on these figures, it seems that theapplication of test automation is perceived as an importantfactor of the test process development by the softwareindustry.

The testing work can be divided into manual testingand automated testing. Automation is usually applied torunning repetitive tasks such as unit testing or regressiontesting, where test cases are executed every time changes aremade [7]. Typical tasks of test automation systems includedevelopment and execution of test scripts and verification oftest results. In contrast to manual testing, automated testingis not suitable for tasks in which there is little repetition [8],such as explorative testing or late development verification


2/18

2 Advances in Software Engineering

tests. For these activities manual testing is more suitable, asbuilding automation is an extensive task and feasible onlyif the case is repeated several times [7, 8]. However, thedivision between automated and manual testing is not asstraightforward in practice as it seems; a large concern is alsothe testability of the software [9], because every piece of code

can be made poorly enough to be impossible to test it reliably,therefore making it ineligible for automation.Software engineering research has two key objectives: the

reduction of costs and the improvement of the quality ofproducts [10]. As software testing represents a significantpart of quality costs, the successful introduction of testautomation infrastructure has a possibility to combine thesetwo objectives, and to overall improve the software testingprocesses. In a similar prospect, the improvements of thesoftware testing processes are also at the focus point of thenew software testing standard ISO 29119 [11]. The objectiveof the standard is to offer a company-level model for thetest processes, offering control, enhancement and follow-upmethods for testing organizations to develop and streamlinethe overall process.

In our prior research project [4, 5, 1214], expertsfrom industry and research institutes prioritized issues ofsoftware testing using the Delphi method [15]. The expertsconcluded that process improvement, test automation withtesting tools, and the standardization of testing are themost prominent issues in concurrent cost reduction andquality improvement. Furthermore, the focused study ontest automation [4] revealed several test automation enablersand disablers which are further elaborated in this study.Our objective is to observe software test automation inpractice, and further discuss the applicability, usability andmaintainability issues found in our prior research. Thegeneral software testing concepts are also observed from theviewpoint of the ISO 29119 model, analysing the test processfactors that create the testing strategy in organizations. Theapproach to achieve these objectives is twofold. First, we wishto explore the software testing practices the organizations areapplying and clarify the current status of test automationin the software industry. Secondly, our objective is toidentify improvement needs and suggest improvements forthe development of software testing and test automationin practice. By understanding these needs, we wish togive both researchers and industry practitioners an insightinto tackling the most hindering issues while providingsolutions and proposals for software testing and automation

improvements.The study is purely empirical and based on observa-

tions from practitioner interviews. The interviewees of thisstudy were selected from companies producing softwareproducts and applications at an advanced technical level.The study included three rounds of interviews and aquestionnaire, which was filled during the second interviewround. We personally visited 31 companies and carried out55 structured or semistructured interviews which were tape-recorded for further analysis. The sample selection aimed torepresent different polar points of the software industry; theselection criteria were based on concepts such as operatingenvironments, product and application characteristics (e.g.,

criticality of the products and applications, real time opera-tion), operating domain and customer base.

The paper is structured as follows. First, in Section 2we introduce comparable surveys and related research.Secondly, the research process and the qualitative andquantitative research methods are described in Section 3.

Then the survey results are presented in Section 4 andthe interview results are presented in Section 5. Finally,the results and observations and their validity are dis-cussed in Section 6 and closing conclusions are discussed inSection 7.

2. Related Research

Besides our prior industry-wide research in testing [4, 5, 1214], software testing practices and test process improvementhave also been studied by others, like Ng et al. [16]in Australia. Their study applied the survey method toestablish knowledge on such topics as testing methodologies,tools, metrics, standards, training and education. The studyindicated that the most common barrier to developingtesting was the lack of expertise in adopting new testingmethods and the costs associated with testing tools. Intheir study, only 11 organizations reported that they mettesting budget estimates, while 27 organizations spent 1.5times the estimated cost in testing, and 10 organizationseven reported a ratio of 2 or above. In a similar vein,Torkar and Mankefors [17] surveyed different types ofcommunities and organizations. They found that 60% ofthe developers claimed that verification and validation werethe first to be neglected in cases of resource shortagesduring a project, meaning that even if the testing isimportant part of the project, it usually is also the firstpart of the project where cutbacks and downscaling areapplied.

As for the industry studies, a similar study approachhas previously been used in other areas of software engi-neering. For example, Ferreira and Cohen [18] completeda technically similar study in South Africa, although theirstudy focused on the application of agile development andstakeholder satisfaction. Similarly, Li et al. [19] conductedresearch on the COTS-based software development processin Norway, and Chen et al. [20] studied the applicationof open source components in software development inChina. Overall, case studies covering entire industry sectorsare not particularly uncommon [21, 22]. In the context of

test automation, there are several studies and reports intest automation practices (such as [2326]). However, thereseems to be a lack of studies that investigate and compare thepractice of software testing automation in different kinds ofsoftware development organizations.

In the process of testing software for errors, testingwork can be roughly divided into manual and automatedtesting, which both have individual strengths and weak-nesses. For example, Ramler and Wolfmaier [3] summarizethe difference between manual and automated testing bysuggesting that automation should be used to prevent furthererrors in working modules, while manual testing is bettersuited for finding new and unexpected errors. However, how


3/18

Advances in Software Engineering 3

and where the test automation should be used is not sostraightforward issue, as the application of test automationseems to be a strongly diversified area of interest. Theapplication of test automation has been studied for examplein test case generation [27, 28], GUI testing [29, 30] andworkflow simulators [31, 32] to name a few areas. Also

according to Bertolino [33], test automation is a significantarea of interest in current testing research, with a focuson improving the degree of automation by developingadvanced techniques for generating the test inputs, or byfinding support procedures such as error report generationto ease the supplemental workload. According to the samestudy, one of the dreams involving software testing is 100%automated testing. However, for example Bachs [23] studyobserves that this cannot be achieved, as all automationultimately requires human intervention, if for nothing else,then at least to diagnose results and maintain automationcases.

The pressure to create resource savings are in manycase the main argument for test automation. A simple andstraightforward solution for building automation is to applytest automation just on the test cases and tasks that werepreviously done manually [8]. However, this approach isusually unfeasible. As Persson and Yilmazturk [26] note,the establishment of automated testing is a costly, high riskproject with several real possibilities for failure, commonlycalled as pitfalls. One of the most common reasons whycreating test automation fails, is that the software is notdesigned and implemented for testability and reusability,which leads to architecture and tools with low reusabilityand high maintenance costs. In reality, test automation setsseveral requisites on a project and has clear enablers anddisablers for its suitability [4, 24]. In some reported cases[27, 34, 35], it was observed that the application of testautomation with an ill-suited process model may be evenharmful to the overall process in terms of productivity orcost-effectiveness.

Models for estimating testing automation costs, forexample by Ramler and Wolfmaier [3], support decision-making in the tradeoff between automated and manualtesting. Berner et al. [8] also estimate that most of the testcases in one project are run at least five times, and onefourth over 20 times. Therefore the test cases, which are doneconstantly like smoke tests, component tests and integrationtests, seem like ideal place to build test automation. In anycase, there seems to be a consensus that test automation

is a plausible tool for enhancing quality, and consequently,reducing the software development costs in the long run ifused correctly.

Our earlier research on the software test automation [4]has established that test automation is not as straightforwardto implement as it may seem. There are several characteristicswhich enable or disable the applicability of test automation.In this study, our decision was to study a larger group ofindustry organizations and widen the perspective for furtheranalysis. The objective is to observe, how the companies haveimplemented test automation and how they have respondedto the issues and obstacles that affect its suitability in practice.Another objective is to analyze whether we can identify new

kind of hindrances to the application of test automationand based on these findings, offer guidelines on whataspects should be taken into account when implementing testautomation in practice.

3. Research Process

3.1. Research Population and Selection of the Sample. Thepopulation of the study consisted of organization units(OUs). The standard ISO/IEC 15504-1 [36] specifies anorganizational unit (OU) as a part of an organization that isthe subject of an assessment. An organizational unit deploysone or more processes that have a coherent process contextand operates within a coherent set of business goals. Anorganizational unit is typically part of a larger organization,although in a small organization, the organizational unit maybe the whole organization.

The reason to use an OU as the unit for observationwas that we wanted to normalize the effect of the company

size to get comparable data. The initial population andpopulation criteria were decided based on the prior researchon the subject. The sample for the first interview roundconsisted of 12 OUs, which were technically high level units,professionally producing software as their main process. Thissample also formed the focus group of our study. Otherselection criteria for the sample were based on the polar typeselection [37] to cover different types of organizations, forexample different businesses, different sizes of the company,and different kinds of operation. Our objective of using thisapproach was to gain a deep understanding of the cases andto identify, as broadly as possible, the factors and features thathave an effect on software testing automation in practice.

For the second round and the survey, the sample wasexpanded by adding OUs to the study. Selecting the samplewas demanding because comparability is not determinedby the company or the organization but by comparableprocesses in the OUs. With the help of national and localauthorities (the network of the Finnish Funding Agency forTechnology and Innovation) we collected a population of 85companies. Only one OU from each company was acceptedto avoid the bias of over-weighting large companies. EachOU surveyed was selected from a company according tothe population criteria. For this round, the sample size wasexpanded to 31 OUs, which also included the OUs fromthe first round. The selection for expansion was based onprobability sampling; the additional OUs were randomly

entered into our database, and every other one was selectedfor the survey. In the third round, the same sample asin the first round was interviewed. Table 1 introduces thebusiness domains, company sizes and operating areas of ourfocus OUs. The company size classification is taken from[38].

3.2. Interview Rounds. The data collection consisted ofthree interview rounds. During the first interview round,the designers responsible for the overall software structureand/or module interfaces were interviewed. If the OU didnot have separate designers, then the interviewed personwas selected from the programmers based on their role in


4/18


Table 1: Description of the interviewed focus OUs (see also the appendix).

OU Business Company size1/Operation

Case A MES1 producer and electronics manufacturer Small/National

Case B Internet service developer and consultant Small/National

Case C Logistics software developer Large/National

Case D ICT consultant Small/NationalCase E Safety and logistics system developer Medium/National

Case F Naval software system developer Medium/International

Case G Financial software developer Large/National

Case H MES1 producer and logistics service systems provider Medium/International

Case I SME2 business and agriculture ICT service provider Small/National

Case J Modeling software developer Large/International

Case K ICT developer and consultant Large/International

Case L Financial software developer Large/International1

Manufacturing Execution System; 2Small and Medium-sized Enterprise, definition [38].

the process to match as closely as possible to the desired

responsibilities. The interviewees were also selected so thatthey came from the same project, or from positions wherethe interviewees were working on the same product. Theinterviewees were not specifically told not to discuss theinterview questions together, but this behavior was notencouraged either. In a case where an interviewee asked forthe questions or interview themes beforehand, the personwas allowed access to them in order to prepare for themeeting. The interviews in all three rounds lasted about anhour and had approximately 20 questions related to the testprocesses or test organizations. In two interviews, there wasalso more than one person present.

The decision to interview designers in the first round was

based on the decision to gain a better understanding on thetest automation practice and to see whether our hypothesisbased on our prior studies [4, 5, 1214] and supplementingliterature review were still valid. During the first interviewround, we interviewed 12 focus OUs, which were selected torepresent different polar types in the software industry. Theinterviews contained semi-structured questions and weretape-recorded for qualitative analysis. The initial analysisof the first round also provided ingredients for the furtherelaboration of important concepts for the latter rounds. Theinterview rounds and the roles of the interviewees in the caseOUs are described in Table 2.

The purpose of the second combined interview and

survey round was to collect multiple choice survey data andanswers to open questions which were based on the firstround interviews. These interviews were also tape-recordedfor the qualitative analysis of the focus OUs, although themain data collection method for this round was a structuredsurvey. In this round, project or testing managers from31 OUs, including the focus OUs, were interviewed. Theobjective was to collect quantitative data on the softwaretesting process, and further collect material on differenttesting topics, such as software testing and development. Thecollected survey data could also be later used to investigateobservations made from the interviews and vice versa, assuggested in [38]. Managers were selected for this round,

as they tend to have more experience on software projects,

and have a better understanding of organizational andcorporation level concepts and the overall software processbeyond project-level activities.

The interviewees of the third round were testers or, if theOU did not have separate testers, programmers who wereresponsible for the higher-level testing tasks. The interviewsin these rounds were also semi-structured and concerned thework of the interviewees, problems in testing (e.g., increasingcomplexity of the systems), the use of software components,the influence of the business orientation, testing resources,tools, test automation, outsourcing, and customer influencefor the test processes.

The themes in the interview rounds remained similar, but

the questions evolved from general concepts to more detailedones. Before proceeding to the next interview round, allinterviews with the focus OUs were transcribed and analyzedbecause new understanding and ideas emerged during thedata analysis. This new understanding was reflected in thenext interview rounds. The themes and questions for eachof the interview rounds can be found on the project websitehttp://www2.it.lut.fi/project/MASTO/.

3.3. Grounded Analysis Method. The grounded analysiswas used to provide further insight into the softwareorganizations, their software process and testing policies.By interviewing people of different positions from the

production organization, the analysis could gain additionalinformation on testing- and test automation-related con-cepts like different testing phases, test strategies, testing toolsand case selection methods. Later this information couldbe compared between organizations, allowing hypotheseson test automation applicability and the test processesthemselves.

The grounded theory method contains three data analy-sis steps: open coding, axial coding and selective coding. Theobjective for open coding is to extract the categories from thedata, whereas axial coding identifies the connections betweenthe categories. In the third phase, selective coding, the corecategory is identified and described [39]. In practice, these


5/18


Table 2: Interviewee roles and interview rounds.

Round type Number of interviews Interviewee role Description

(1) Semistructured 12 focus OUsDesigner or

ProgrammerThe interviewee is responsible for software designor has influence on how software is implemented

(2) Structured/

Semistructured

31 OUs quantitative,including 12 focus

OUs qualitative

Project or testing

manager

The interviewee is responsible for softwaredevelopment projects or test processes of software

products

(3) Semistructured 12 focus OUs TesterThe interviewee is a dedicated software tester or isresponsible for testing the software product

steps overlap and merge because the theory developmentprocess proceeds iteratively. Additionally, Strauss and Corbin[40] state that sometimes the core category is one of theexisting categories, and at other times no single category isbroad enough to cover the central phenomenon.

The objective of open coding is to classify the data intocategories and identify leads in the data, as shown in Table 3.The interview data is classified to categories based on themain issue, with observation or phenomenon related to itbeing the codified part. In general, the process of groupingconcepts that seem to pertain to the same phenomena iscalled categorizing, and it is done to reduce the numberof units to work with [40]. In our study, this was doneusing ATLAS.ti software [41]. The open coding processstarted with seed categories [42] that were formed fromthe research question and objectives, based on the literaturestudy on software testing and our prior observations [4, 5,1214] on software and testing processes. Some seed cate-gories, like knowledge management, service-orientationor approach for software development were derived from

our earlier studies, while categories like strategy for testing,outsourcing, customer impact or software testing toolswere taken from known issues or literature review observa-tions.

The study followed the approach introduced by Seaman[43], which notes that the initial set of codes (seed categories)comes from the goals of the study, the research questions, andpredefined variables of interest. In the open coding, we addednew categories and merged existing categories to others ifthey seemed unfeasible or if we found a better generalization.Especially at the beginning of the analysis, the numberof categories and codes quickly accumulated and the totalnumber of codes after open coding amounted to 164 codes

in 12 different categories. Besides the test process, softwaredevelopment in general and test automation, these categoriesalso contained codified observations on such aspects as thebusiness orientation, outsourcing, and product quality.

After collecting the individual observations to categoriesand codes, the categorized codes were linked together basedon the relationships observed in the interviews. For example,the codes Software process: Acquiring 3rd party modules,Testing strategy: Testing 3rd party modules, and Problem:Knowledge management with 3rd party modules wereclearly related and therefore we connected them togetherin axial coding. The objective of axial coding is to furtherdevelop categories, their properties and dimensions, and find

causal, or any kinds of, connections between the categoriesand codes.

For some categories, the axial coding also makes it pos-sible to define dimension for the phenomenon, for examplePersonification-Codification for Knowledge managementstrategy, where every property could be defined as a pointalong the continuum defined by the two polar opposites.For the categories that are given dimension, the dimensionrepresented the locations of the property or the attributeof a category [40]. Obviously for some categories, whichwere used to summarize different observations like enhance-ment proposals or process problems, defining dimensionswas unfeasible. We considered using dimensions for somecategories like criticality of test automation in testingprocess or tool sophistication level for automation toolsin this study, but discarded them later as they yielded onlylittle value to the study. This decision was based on theobservation that values of both dimensions were outcomesof the applied test automation strategy, having no effect onthe actual suitability or applicability of test automation to the

organizations test process.Our approach for analysis of the categories included

Within-Case Analysis and Cross-Case-Analysis, as speci-fied by Eisenhardt [37]. We used the tactic of selectingdimensions and properties with within-group similaritiescoupled with inter-group differences [37]. In this strategy,our team isolated one phenomenon that clearly dividedthe organizations to different groups, and searched forexplaining differences and similarities from within thesegroups. Some of the applied features were, for example, theapplication of agile development methods, the applicationof test automation, the size [38] difference of originatingcompanies and service orientation. As for one central

result, the appropriateness of OU as a comparison unit wasconfirmed based on our size difference-related observationson the data; the within-group- and inter-group comparisonsdid yield results in which the company size or companypolicies did not have strong influence, whereas the local,within-unit policies did. In addition, the internal activitiesobserved in OUs were similar regardless of the originatingcompany size, meaning that in our study the OU comparisonwas indeed feasible approach.

We established and confirmed each chain of evidence inthis interpretation method by discovering sufficient citationsor finding conceptually similar OU activities from the casetranscriptions. Finally, in the last phase of the analysis,


6/18


Table 3: Open coding of the interview data.

Interview transcript Codes, Category: Code

Well, I would hope for stricter control or management for implementing our testing strategy, as Iam not sure if our testing covers everything and is it sophisticated enough . On the other hand, wedo have strictly limited resources, so it can be enhanced only to some degree, we cannot testeverything. And perhaps, recently we have had, in the newest versions, some regression testing,going through all features, seeing if nothing is broken, but in several occasions this has been leftunfinished because time has run out. So there, on that issue we should focus.

Enhancement proposal:Developing testing strategy

Strategy for testing: Ensuringcase coverage

Problem: Lack of resources

Problem: Lack of time

in selective coding, our objective was to identify the corecategory [40]a central phenomenonand systematicallyrelate it to other categories and generate the hypothesis andthe theory. In this study, we consider test automation in prac-tice as the core category, to which all other categories wererelated as explaining features of applicability or feasibility.

The general rule in grounded theory is to sample untiltheoretical saturation is reached. This means (1) no newor relevant data seem to emerge regarding a category, (2)the category development is dense, insofar as all of theparadigm elements are accounted for, along with variationand process, and (3) the relationships between categoriesare well established and validated [40]. In our study, thesaturation was reached during the third round, where nonew categories were created, merged, or removed fromthe coding. Similarly, the attribute values were also stable,that is, the already discovered phenomena began to repeatthemselves in the collected data. As an additional wayto ensure the validity of our study and avoid validitythreats [44], four researchers took part in the data analysis.The bias caused by researchers was reduced by combiningthe different views of the researchers (observer triangu-

lation) and a comparison with the phenomena observedin the quantitative data (methodological triangulation)[44, 45].

3.4. The Survey Instrument Development and Data Collection.The survey method described by Fink and Kosecoff [46]was used as the research method in the second round.An objective for a survey is to collect information frompeople about their feelings and beliefs. Surveys are mostappropriate when information should come directly fromthe people [46]. Kitchenham et al. [47] divide comparablesurvey studies into exploratory studies from which only weakconclusions can be drawn, and confirmatory studies from

which strong conclusions can be drawn. We consider thisstudy as an exploratory, observational, and cross-sectionalstudy that explores the phenomenon of software testingautomation in practice and provides more understanding toboth researchers and practitioners.

To obtain reliable measurements in the survey, a vali-dated instrument was needed, but such an instrument wasnot available in the literature. However, Dyba [48] hasdeveloped an instrument for measuring the key factors ofsuccess in software process improvement. Our study wasconstructed based on the key factors of this instrument, andsupplemented with components introduced in the standardsISO/IEC 29119 [11] and 25010 [49]. We had the possibility

to use the current versions of the new standards because oneof the authors is a member of the JTC1/SC7/WG26, whichis developing the new software testing standard. Based onthese experiences a measurement instrument derived fromthe ISO/IEC 29119 and 25010 standards was used.

The survey consisted of a questionnaire (available athttp://www2.it.lut.fi/project/MASTO/) and a face-to-faceinterview. Selected open-ended questions were located at theend of the questionnaire to cover some aspects related to ourqualitative study. The classification of the qualitative answerswas planned in advance.

The questionnaire was planned to be answered duringthe interview to avoid missing answers because they makethe data analysis complicated. All the interviews were tape-recorded, and for the focus organizations, further quali-tatively analyzed with regard to the additional commentsmade during the interviews. Baruch [50] also states thatthe average response rate for self-assisted questionnaires is55.6%, and when the survey involves top management ororganizational representatives the response rate is 36.1%. Inthis case, a self-assisted, mailed questionnaire would haveled to a small sample. For these reasons, it was rejected, and

personal interviews were selected instead. The questionnairewas piloted with three OUs and four private persons.

If an OU had more than one respondent in the inter-view, they all filled the same questionnaire. Arranging theinterviews, traveling and interviewing took two months ofcalendar time. Overall, we were able to accomplish 0.7 surveyinterviews per working day on an average. One researcherconducted 80% of the interviews, but because of theoverlapping schedules also two other researchers participatedin the interviews. Out of the contacted 42 OUs, 11 wererejected because they did not fit the population criteria inspite of the source information, or it was impossible to fit theinterview into the interviewees schedule. In a few individual

cases, the reason for rejection was that the organizationrefused to give an interview. All in all, the response rate was,therefore, 74%.

4. Testing and Test Automation inSurveyed Organizations

4.1. General Information of the Organizational Units. Theinterviewed OUs were parts of large companies (55%)and small and medium-sized enterprises (45%). The OUsbelonged to companies developing information systems(11 OUs), IT services (5 OUs), telecommunication (4 OUs),


7/18


1

12

3

4

4

5

11

Public sector

LogisticsMetal industry

Ind. automation

Telecommunications

Finances

IT services

IT development

Figure 1: Application domains of the companies.

Table 4: Interviewed OUs.

Max. Min. Median

Number of employees inthe company.

350 000 4 315

Number of SW developersand testers in the OU.

600 01 30

Percentage of automationin testing.

90 0 10

Percentage of agile(reactive, iterative) versusplan driven methods inprojects.

100 0 30

Percentage of existingtesters versus resourcesneed.

100 10 75

How many percent of thedevelopment effort is spenton testing?

70 02 25

10 means that all of the OUs developers and testers are acquired from 3rd

parties.20 means that no project time is allocated especially for testing.

finance (4 OUs), automation systems (3 OUs), the metalindustry (2 OUs), the public sector (1 OU), and logistics(1 OU). The application domains of the companies arepresented in Figure 1. Software products represented 63% ofthe turnover, and services (e.g., consulting, subcontracting,and integration) 37%.

The maximum number of personnel in the companiesto which the OUs belonged was 350 000, the minimum wasfour, and the median was 315. The median of the software

developers and testers in the OUs was 30 persons. OUsapplied automated testing less than expected, the median ofthe automation in testing being 10%. Also, the interviewedOUs utilized agile methods less than expected: the medianof the percentage of agile (reactive, iterative) versus plandriven methods in projects was 30%. The situation withhuman resources was better than what was expected, asthe interviewees estimated that the amount of humanresources in testing was 75%. When asking what percentof the development effort was spent on testing, the medianof the answers was 25%. The cross-sectional situation ofdevelopment and testing in the interviewed OUs is illustratedin Table 4.

Case A

Case B

Case C

Case D

Case E

Case F

Case G

Case H

Case I

Case J

Case K

Case L

Survey average

Percentage of project effort allocated

solely to testingPercentage of tests resources from optimal

amount (has 2 needs 3 equals 66%)Percentage of test automation from all testcases

0 20 40 60 80 100

010

35

6020

606020

2020

25050

1520

7520

5590

3560

301025

7050

075

105 100

3590

702026

7027

Figure 2: Amountof test resources and test automation in the focusorganizations of the study and the survey average.

The amount of testing resources was measured bythree figures; first the interviewee was asked to evaluatethe percentage from total project effort allocated solely totesting. The survey average was 27%, the maximum being70% and the minimum 0%, meaning that the organizationrelied solely on testing efforts carried out in parallel with

development. The second figure was the amount of testresources compared to the organizational optimum. In thisfigure, if the company had two testers and required three,it would have translated as 66% of resources. Here theaverage was 70%; six organizations (19%) reported 100%resource availability. The third figure was the number ofautomated test cases compared to all of the test cases in all ofthe test phases the software goes through before its release.The average was 26%, varying between different types oforganizations and project types. The results are presentedin Figure 2, in which the qualitative study case OUs are alsopresented for comparison. The detailed descriptions for eachcase organization are available in the appendix.

4.2. General Testing Items. The survey interviewed 31 orga-nization managers from different types of software industry.The contributions of the interviewees were measured usinga five-point Likert scale where 1 denoted I fully disagreeand 5 denoted I fully agree. The interviewees emphasizedthat quality is built in development (4.3) rather than intesting (2.9). Then the interviewees were asked to estimatetheir organizational testing practices according to the newtesting standard ISO/IEC 29119 [11], which identifies fourmain levels for testing processes: the test policy, test strategy,test management and testing. The test policy is the companylevel guideline which defines the management, framework


8/18


The OUs test executionis excellent

The OUs test managementis excellent

The OUs test strategyis excellent

The OUs test policy is excellent

Quality is built in testing

Quality is built in development

1 1.5 2 2.5 3 3.5 4 4.5 5

3.

5

3.4

3.3

3.3

2.9

4.3

Figure 3: Levels of testing according to the ISO/IEC 29119standard.

Unit testing is excellent

Integration testing isexcellent

Usability testing is excellent

Functional testing isexcellent

System testing is excellent

Conformance testing isexcellent

1 1.5 2 2.5 3 3.5 4 4.5 5

2.8

3

3.1

3.8

3.6

3.3

Figure 4: Testing phases in the software process.

and general guidelines, the test strategy is an adaptive modelfor the preferred test process, test management is the controllevel for testing in a software project, and finally, testingis the process of conducting test cases. The results did notmake a real difference between the lower levels of testing (testmanagement level and test levels) and higher levels of testing(organizational test policy and organizational test strategy).All in all, the interviewees were rather satisfied with thecurrent organization of testing. The resulted average levelsfrom quantitative survey are presented in Figure 3.

Besides the organization, the test processes and testphases were also surveyed. The five-point Likert scale withthe same one to fiveone being fully disagree and fivefully agreegrading method was used to determine thecorrectness of different testing phases. Overall, the lattertest phasessystem, functional testingwere consideredexcellent or very good, whereas the low level test phasessuch as unit testing and integration received several low-end scores. The organizations were satisfied or indifferenttowards all test phases, meaning that there were no strongfocus areas for test organization development. However,

based on these results it seems plausible that one effectiveway to enhance testing would be to support low-level testingin unit and integration test phases. The results are depictedin Figure 4.

Finally, the organizations surveyed were asked to ratetheir testing outcomes and objectives (Figure 5). The firstthree items discussed the test processes of a typical softwareproject. There seems to be a strong variance in testingschedules and time allocation in the organizations. Theoutcomes 3.2 for schedule and 3.0 for time allocation donot give any information by themselves, and overall, thedirection of answers varied greatly between Fully disagreeand Fully agree. However, the situation with test processes

We have prioritized the most

important quality attributes

We have identified the most

important quality attributes

Testing has enough time

Testing phases are kept

Testing stays in schedule

1 1.5 2 2.5 3 3.5 4 4.5 5

3.3

3.7

3

3.5

3.2

Figure 5: Testing process outcomes.

was somewhat better; the result 3.5 may also not be astrong indicator by itself, but the answers had only littlevariance, 20 OUs answering somewhat agree or neutral.

This indicates that even if the time is limited and the projectschedule restricts testing, the testing generally goes throughthe normal, defined, procedures.

The fourth and fifth items were related to quality aspects,and gave insights into the clarity of testing objectives. Theresults of 3.7 for the identification of quality attributesindicate that organizations tend to have objectives for thetest processes and apply quality criteria in development.However, the prioritization of their quality attributes is notas strong (3.3) as identification.

4.3. Testing Environment. The quality aspects were alsoreflected in the employment of systematic methods for thetesting work. The majority (61%) of the OUs followeda systematic method or process in the software testing,13% followed one partially, and 26% of the OUs did notapply any systematic method or process in testing. Processpractices were derived from, for example, TPI (Test ProcessImprovement) [51] or the Rational Unified Process (RUP)[52]. Few Agile development process methods such asScrum [53] or XP (eXtreme Programming) [54] were alsomentioned.

A systematic method is used to steer the software project,but from the viewpoint of testing, the process also needsan infrastructure on which to operate. Therefore, the OUswere asked to report which kind of testing tools they apply to

their typical software processes. The test management tools,tools which are used to control and manage test cases andallocate testing resources to cases, turned out to be the mostpopular category of tools; 15 OUs out of 31 reported the useof this type of tool. The second in popularity were manualunit testing tools (12 OUs), which were used to execute testcases and collect test results. Following them were tools toimplement test automation, which were in use in 9 OUs,performance testing tools used in 8 OUs, bug reporting toolsin 7 OUs and test design tools in 7 OUs. Test design tools wereused to create and design new test cases. The group of othertools consisted of, for example, electronic measurementdevices, test report generators, code analyzers, and project


9/18


Other

Quality control tools

Test design software

Bug reporting

Performance testing

Test automation

Unit testing

Test case management

10

6

7

7

8

9

12

15

Figure 6: Popularity of the testing tools according to the survey.

management tools. The popularity of the testing tools indifferent survey organizations is illustrated in Figure 6.

The respondents were also asked to name and explainthe three most efficient application areas of test automationtools. Both the format of the open-ended questions and theclassification of the answers were based on the like best (LB)technique adopted from Fink and Kosecoff [46]. According

to the LB technique, respondents were asked to list pointsthey considered the most efficient. The primary selectionwas the area in which the test automation would be themost beneficial to the test organization, the secondary oneis the second best area of application, and the third one isthe third best area. The interviewees were also allowed toname only one or two areas if they were unable to decideon three application areas. The results revealed the relativeimportance of software testing tools and methods.

The results are presented in Figure 7. The answers weredistributed rather evenly between different categories of toolsor methods. The most popular category was unit testing toolsor methods (10 interviewees). Next in line were regression

testing (9), tools to support testability (9), test environ-ment tools and methods (8), and functional testing (7).The group others (11) consisted of conformance testingtools, TTCN-3 (Testing and Test Control Notation version3) tools, general test management tools such as documentgenerators and methods of unit and integration testing. Themost popular category, unit testing tools or methods, alsoreceived the most primary application area nominations. Themost common secondary area of application was regressiontesting. Several categories ranked third, but concepts suchas regression testing, and test environment-related aspectssuch as document generators were mentioned more thanonce. Also testability-related conceptsmodule interface,

conformance testingor functional testingverification,validation testswere considered feasible implementationareas for test automation.

4.4. Summary of the Survey Findings. The survey suggeststhat interviewees were rather satisfied with their test policy,test strategy, test management, and testing, and did nothave any immediate requirements for revising certain testphases, although low-level testing was slightly favoured in thedevelopment needs. All in all, 61% of the software companiesfollowed some form of a systematic process or methodin testing, with an additional 13% using some establishedprocedures or measurements to follow the process efficiency.

Other

Performance testing

Functional testing

Test environment-related

Testability-related

Regression testing

Unit testing

Primary

Secondary

Tertiary

0 2 4 6 8 10 12

2 4 5

2 1

3 3 1

4 2 2

4 4 1

2 5 2

8 2

Figure 7: The three most efficient application areas of testautomation tools according to the interviewees.

The systematic process was also reflected in the generalapproach to testing; even if the time was limited, the testprocess followed a certain path, applying the test phases

regardless of the project limitations.The main source of the software quality was considered

to be in the development process. In the survey, the testorganizations used test automation on an average on 26%of their test cases, which was considerably less than could beexpected based on the literature. However, test automationtools were the third most common category of test-relatedtools, commonly intended to implement unit and regressiontesting. As for the test automation itself, the intervieweesranked unit testing tools as the most efficient tools oftest automation, regression testing being the most commonsecondary area of application.

5. Test Automation Interviews andQualitative Study

Besides the survey, the test automation concepts and appli-cations were analyzed based on the interviews with the focusorganizations. The grounded theory approach was applied toestablish an understanding of the test automation conceptsand areas of application for test automation in industrialsoftware engineering. The qualitative approach was appliedin three rounds, in which a developer, test manager and testerfrom 12 different case OUs were interviewed. Descriptions ofthe case OUs can be found in the appendix.

In theory-creating inductive research [55], the central

idea is that researchers constantly compare theory anddata iterating with a theory which closely fits the data.Based on the grounded theory codification, the categoriesidentified were selected in the analysis based on their abilityto differentiate the case organizations and their potentialto explain the differences regarding the application of testautomation in different contexts. We selected the categoriesso as to explore the types of automation applications andthe compatibility of test automation services with the OUstesting organization. We conceptualized the most commontest automation concepts based on the coding and furtherelaborated them to categories to either cater the essentialfeatures such as their role in the overall software process or


10/18


their relation to test automation. We also concentrated onthe OU differences in essential concepts such as automationtools, implementation issues or development strategies. Thisconceptualization resulted to the categories listed in Table 5.

The category Automation application describes theareas of software development, where test automation was

applied successfully. This category describes the testing activ-ities or phases which apply test automation processes. In thecase where the test organization did not apply automation, orhad so far only tested it for future applications, this categorywas left empty. The application areas were generally gearedtowards regression and stress testing, with few applicationsof functionality and smoke tests in use.

The category Role in software process is related to theobjective for which test automation was applied in softwaredevelopment. The role in the software process describes theobjective for the existence of the test automation infras-tructure; it could, for example, be in quality control, whereautomation is used to secure module interfaces, or in qualityassurance, where the operation of product functionalities isverified. The usual role for the test automation tools wasin quality control and assurance, the level of applicationvarying from third party-produced modules to primaryquality assurance operations. On two occasions, the roleof test automation was considered harmful to the overalltesting outcomes, and on one occasion, the test automationwas considered trivial, with no real return on investmentscompared to traditional manual testing.

The category Test automation strategy is the approachto how automated testing is applied in the typical softwareprocesses, that is, the way the automation was used as apart of the testing work, and how the test cases and overalltest automation strategy were applied in the organization.The level of commitment to applying automation was themain dimension of this category, the lowest level beingindividual users with sporadic application in the softwareprojects, and the highest being the application of automationto the normal, everyday testing infrastructure, where testautomation was used seamlessly with other testing methodsand had specifically assigned test cases and organizationalsupport.

The category of Automation development is the generalcategory for OU test automation development. This categorysummarizes the ongoing or recent efforts and resourceallocations to the automation infrastructure. The type of newdevelopment, introduction strategies and current develop-

ment towards test automation are summarized in this cate-gory. The most frequently chosen code was general increaseof application, where the organization had committed itselfto test automation, but had no clear idea of how to developthe automation infrastructure. However, one OU had adevelopment plan for creating a GUI testing environment,while two organizations had just recently scaled down theamount of automation as a result of a pilot project. Twoorganizations had only recently introduced test automationto their testing infrastructure.

The category of Automation tools describes the typesof test automation tools that are in everyday use in theOU. These tools are divided based on their technological

finesse, varying from self-created drivers and stubs toindividual proof-of-concept tools with one specified taskto test suites where several integrated components are usedtogether for an effective test automation environment. Ifthe organization had created the tools by themselves, orcustomized the acquired tools to the point of having new

features and functionalities, the category was supplementedwith a notification regarding in-house-development.Finally, the category of Automation issues includes

the main hindrances which are faced in test automationwithin the organization. Usually, the given issue was relatedto either the costs of test automation or the complexityof introducing automation to the software projects whichhave been initially developed without regards to supportfor automation. Some organizations also considered theefficiency of test automation to be the main issue, mostlycontributing to the fact that two of them had just recentlyscaled down their automation infrastructure. A complete listof test automation categories and case organizations is givenin Table 6.

We elaborated further these properties we observedfrom the case organizations to create hypotheses for thetest automation applicability and availability. These result-ing hypotheses were shaped according to advice given byEisenhardt [37] for qualitative case studies. For example,we perceived the quality aspect as really important forthe role of automation in software process. Similarly, theresource needs, especially costs, were much emphasizedin the automation issues category. The purpose of thehypotheses below is to summarize and explain the featuresof test automation that resulted from the comparison ofdifferences and similarities between the organizations.

Hypothesis 1 (Test automation should be considered moreas a quality control tool rather than a frontline testingmethod). The most common area of application observedwas functionality verification, that is, regression testing andGUI event testing. As automation is time-consuming andexpensive to create, these were the obvious ways to createtest cases which had the minimal number of changes perdevelopment cycle. By applying this strategy, organizationscould set test automation to confirm functional propertieswith suitable test cases, and acquire such benefits as supportfor change management and avoid unforeseen compatibilityissues with module interfaces.

Yes, regression testing, especially automated. Itis not manually hammered in every time, butused so that the test sets are run, and if thereis anything abnormal, it is then investigated.Manager, Case G

. . . had we not used it [automation tests], itwould have been suicidal.Designer, Case D

Its [automated stress tests] good for showing badcode, how efficient it is and how well designed . . .stress it enough and we can see if it slows down oreven breaks completely.Tester, Case E


11/18


Table 5: Test automation categories.

Category Definition

Automation application Areas of application for test automation in the software process

Role in software processThe observed roles of test automation in the company software process andthe effect of this role

Test automation strategy The observed method for selecting the test cases where automation isapplied and the level of commitment to the application of test automationin the organizations

Automation developmentThe areas of active development in which the OU is introducing testautomation

Automation tools The general types of test automation tools applied

Automation issues The items that hinder test automation development in the OU

However, there seemed to be some contradicting con-siderations regarding the applicability of test automation.Cases F, J, and K had recently either scaled down their testautomation architecture or considered it too expensive orinefficient when compared to manual testing. In some cases,automation was also considered too bothersome to configurefor a short-term project, as the system would have requiredconstant upkeep, which was an unnecessary addition to theproject workload.

We really have not been able to identifyany major advancements from it [test automa-tion].Tester, Case J

It [test automation] just kept interfering.Designer, Case K

Both these viewpoints indicated that test automationshould not be considered a frontline test environment for

finding errors, but rather a quality control tool to maintainfunctionalities. For unique cases or small projects, testautomation is too expensive to develop and maintain, andit generally does not support single test cases or explorativetesting. However, it seems to be practical in larger projects,where verifying module compatibility or offering legacysupport is a major issue.

Hypothesis 2 (Maintenance and development costs are com-mon test automation hindrances that universally affect alltest organizations regardless of their business domain orcompany size). Even though the case organizations wereselected to represent different types of organizations, the

common theme was that the main obstacles in automationadoption were development expenses and upkeep costs. Itseemed to make no difference whether the organization unitbelonged to a small or large company, as in the OU levels theyshared common obstacles. Even despite the maintenanceand development hindrances, automation was considered afeasible tool in many organizations. For example, Cases I andL pursued the development of some kind of automation toenhance the testing process. Similarly, Cases E and H, whichalready had a significant number of test automation cases,were actively pursuing a larger role for automated testing.

Well, it [automation] creates a sense of securityand controllability, and one thing that is easily

underestimated is its effect on performance andoptimization. It requires regression tests to confirmthat if something is changed, the whole thing doesnot break down afterwards.Designer, Case H

In many cases, the major obstacle for adopting testautomation was, in fact, the high requirements for processdevelopment resources.

Shortage of time, resources . . . we have thetechnical ability to use test automation, but wedont. Tester, Case J

Creating and adopting it, all that it takes to makeusable automation . . . I believe that we dont putany effort into it because it will end up being reallyexpensive. Designer, Case J

In Case J particularly, the OU saw no incentive indeveloping test automation as it was considered to offer onlylittle value over manual testing, even if they otherwise had noimmediate obstacles other than implementation costs. Alsocases F and K reported similar opinions, as they both hadscaled down the amount of automation after the initial pilotprojects.

It was a huge effort to manually confirm why theresults were different, so we took it [automation]down.Tester, Case F

Well, we had gotten automation tools from ourpartner, but they were so slow we decided to go on

with manual testing.Tester, Case K

Hypothesis 3 (Test automation is applicable to most of thesoftware processes, but requires considerable effort from theorganization unit). The case organizations were selected torepresent the polar types of software production operatingin different business domains. Out of the focus OUs, therewere four software development OUs, five IT service OUs,two OUs from the finance sector and one logistics OU. Ofthese OUs, only two did not have any test automation, andtwo others had decided to strategically abandon their testautomation infrastructure. Still, the business domains for theremaining organizations which applied test automation were


12/18


Table 6: Test automation categories affecting the software process in case OUs.

OUCategory

Automationapplication

Role in softwareprocess

Test automationstrategy

Automationdevelopment

Automationtools

Automationissues

Case A

GUI testing,

regressiontesting

Functionality

verification

Part of the

normal testinfrastructure

General

increase ofapplication

Individual tools,test suite,

in-housedevelopment

Complexity ofadapting

automation totest processes

Case BPerformance,smoke testing

Quality controltool

Part of thenormal testinfrastructure

GUI testing,unit testing

Individual tools,in-housedevelopment

Costs ofautomationimplementation

Case C

Functionality,regressiontesting,documentationautomation

Quality controltool


Generalincrease ofapplication

Test suite,in-housedevelopment

Cost ofautomationmaintenance

Case DFunctionalitytesting

Quality controlfor secondarymodules

Project-relatedcases

Upkeep forexisting parts

Individual toolsCosts ofautomationimplementation

Case ESystem stresstesting

Qualityassurance tool



Test suiteCosts ofimplementingnew automation

Case F

Unit andmodule testing,documentationautomation

QC, overalleffect harmful

Individual usersRecently scaleddown

Individual toolsManual testingseen moreefficient

Case GRegressiontesting for usecases

Qualityassurance tool



Test suiteCost ofautomationmaintenance

Case H

Regressiontesting formoduleinterfaces

Quality controlfor secondarymodules



Test suite,in-housedevelopment

Underestimationof the effect ofautomated testingon quality

Case IFunctionalitytesting

Quality controltool


Applicationpilot indevelopment

Proof-of-concepttools

Costs ofautomationimplementation

Case JAutomation notin use

QA, no effectobserved

Individual usersApplicationpilot indevelopment

Proof-of-concepttools

No developmentincentive

Case KSmall scalesystem testing

QC, overalleffect harmful

Individual usersRecently scaleddown

Self-createdtools; driversand stubs

Manual testingseen moreefficient

Case LSystem stresstesting

Verifies modulecompatibility


Adaptingautomation tothe testingstrategy

Individual tools,in-housedevelopment

Complexity ofadaptingautomation totest processes

heterogeneously divided, meaning that the business domainis not a strong indicator of whether or not test automationshould be applied.

It seems that test automation is applicable as a test tool inany software process, but the amount of resources requiredfor useful automation compared to the overall developmentresources is what determines whether or not automationshould be used. As automation is oriented towards qualitycontrol aspects, it may be unfeasible to implement in smalldevelopment projects where quality control is manageablewith manual confirmation. This is plausible, as the amount

of required resources does not seem to vary based onaspects beyond the OU characteristics, such as availablecompany resources or testing policies applied. The feasibilityof test automation seems to be rather connected to theactual software process objectives, and fundamentally to thedecision whether the quality control aspects gained from testautomation supersede the manual effort required for similarresults.

. . . before anything is automated, we shouldcalculate the maintenance effort and estimate


13/18


whether we will really save time, instead of justautomating for automations sake.Tester, CaseG

It always takes a huge amount of resources toimplement.Designer, Case A

Yes, developing that kind of test automationsystem is almost as huge an effort as building theactual project.Designer, Case I

Hypothesis 4 (The available repertoire of testing automationtools is limited, forcing OUs to develop the tools them-selves, which subsequently contributes to the applicationand maintenance costs). There were only a few case OUsthat mentioned any commercial or publicly available testautomation programs or suites. The most common approachto test automation tools was to first acquire some sortof tool for proof-of-concept piloting, then develop similartools as in-house-production or extend the functionalities

beyond the original tool with the OUs own resources. Theseresources for in-house-development and upkeep for self-made products are one of the components that contributeto the costs of applying and maintaining test automation.

Yes, yes. That sort of [automation] tools havebeen used, and then theres a lot of work that we doourselves. For example, this stress test tool . . .Designer, Case E

We have this 3rd party library for the automa-tion. Well, actually, we have created our ownarchitecture on top of it. . .Designer, Case H

Well, in [company name], weve-, we developedour own framework to, to try and get around someof these, picking which tests, which group of testsshould be automated.Designer, Case C

However, it should be noted that even if the automationtools were well-suited for the automation tasks, the main-tenance still required significant resources if the softwareproduct to which it was connected was developing rapidly.

Well, there is the problem [with automation tool]that sometimes the upkeep takes an incrediblylarge amount of time.Tester, Case G

Our system keeps constantly evolving, so youdhave to be constantly recording [maintainingtools]. . .Tester, Case K

6. Discussion

An exploratory survey combined with interviews was usedas the research method. The objective of this study was toshed light on the status of test automation and to identifyimprovement needs in and the practice of test automation.The survey revealed that the total effort spent on testing(median 25%) was less than expected. The median percent-age (25%) of testing is smaller than the 50%60% that is

often mentioned in the literature [38, 39]. The comparablelow percentage may indicate that that the resources neededfor software testing are still underestimated even thoughtesting efficiency has grown. The survey also indicated thatcompanies used fewer resources on test automation thanexpected: on an average 26% of all of the test cases apply

automation. However, there seems to be ambiguity as towhich activities organizations consider test automation, andhow automation should be applied in the test organizations.In the survey, several organizations reported that they havean extensive test automation infrastructure, but this didnot reflect on the practical level, as in the interviews withtesters particularly, the figures were considerably different.This indicates that the test automation does not havestrong strategy in the organization, and has yet to reachmaturity in several test organizations. Such concepts asquality assurance testing and stress testing seem to beparticularly unambiguous application areas, as Cases E and Ldemonstrated. In Case E, the management did not considerstress testing an automation application, whereas testers did.Moreover, in Case L the large automation infrastructuredid not reflect on the individual project level, meaningthat the automation strategy may strongly vary betweendifferent projects and products even within one organizationunit.

The qualitative study which was based on interviewsindicated that some organizations, in fact, actively avoidusing test automation, as it is considered to be expensiveand to offer only little value for the investment. However,test automation seems to be generally applicable to thesoftware process, but for small projects the investment isobviously oversized. One additional aspect that increasesthe investment are tools, which unlike in other areas ofsoftware testing, tend to be developed in-house or areheavily modified to suit specific automation needs. Thisdevelopment went beyond the localization process whichevery new software tool requires, extending even to thedevelopment of new features and operating frameworks. Inthis context it also seems plausible that test automationcan be created for several different test activities. Regressiontesting, GUI testing or unit testing, activities which in someform exist in most development projects, all make it possibleto create successful automation by creating suitable tools forthe task, as in each phase can be found elements that havesufficient stability or unchangeability. Therefore it seems thatthe decision on applying automation is not only connected to

the enablers and disablers of test automation [4], but ratheron tradeoffof required effort and acquired benefits; In smallprojects or with low amount of reuse the effort becomestoo much for such investment as applying automation to befeasible.

The investment size and requirements of the effortcan also be observed on two other occasions. First, testautomation should not be considered as an active testing toolfor finding errors, but as a tool to guarantee the functionalityof already existing systems. This observation is in line withthose of Ramler and Wolfmaier [3], who discuss the necessityof a large number of repetitive tasks for the automationto supersede manual testing in cost-effectiveness, and of


14/18


Berner et al. [8], who notify that the automation requiresa sound application plan and well-documented, simulatableand testable objects. For both of these requirements, qualitycontrol at module interfaces and quality assurance on systemoperability are ideal, and as it seems, they are the mostcommonly used application areas for test automation. In

fact, Kaner [56] states that 60%80% of the errors found withtest automation are found in the development phase for thetest cases, further supporting the quality control aspect overerror discovery.

Other phenomena that increase the investment are thelimited availability and applicability of automation tools.On several occasions, the development of the automationtools was an additional task for the automation-buildingorganization that required the organization to allocate theirlimited resources to the test automation tool implemen-tation. From this viewpoint it is easy to understand whysome case organizations thought that manual testing issufficient and even more efficient when measured in resourceallocation per test case. Another approach which couldexplain the observed resistance to applying or using testautomation was also discussed in detail by Berner et al. [ 8],who stated that organizations tend to have inappropriatestrategies and overly ambitious objectives for test automationdevelopment, leading to results that do not live up to theirexpectations, causing the introduction of automation tofail. Based on the observations regarding the developmentplans beyond piloting, it can also be argued that the lack ofobjectives and strategy also affect the successful introductionprocesses. Similar observations of automation pitfalls werealso discussed by Persson and Yilmazturk [26] and Mosleyand Posey [57].

Overall, it seems that the main disadvantages of testingautomation are the costs, which include implementationcosts, maintenance costs, and training costs. Implementationcosts included direct investment costs, time, and humanresources. The correlation between these test automationcosts and the effectiveness of the infrastructure are discussedby Fewster [24]. If the maintenance of testing automationis ignored, updating an entire automated test suite can costas much, or even more than the cost of performing allthe tests manually, making automation a bad investmentfor the organization. We observed this phenomenon intwo case organizations. There is also a connection betweenimplementation costs and maintenance costs [24]. If thetesting automation system is designed with the minimization

of maintenance costs in mind, the implementation costsincrease, and vice versa. We noticed the phenomenon ofcosts preventing test automation development in six cases.The implementation of test automation seems to be possibleto accomplish with two different approaches: by promotingeither maintainability or easy implementation. If the selectedfocus is on maintainability, test automation is expensive, butif the approach promotes easy implementation, the processof adopting testing automation has a larger possibility forfailure. This may well be due to the higher expectations andassumption that the automation could yield results fasterwhen promoting implementation over maintainability, oftenleading to one of the automation pitfalls [26] or at least a low

percentage of reusable automation components with highmaintenance costs.

7. Conclusions

The objective of this study was to observe and identify

factors that aff

ect the state of testing, with automation as thecentral aspect, in different types of organizations. Our studyincluded a survey in 31 organizations and a qualitative studyin 12 focus organizations. We interviewed employees fromdifferent organizational positions in each of the cases.

This study included follow-up research on prior obser-vations [4, 5, 1214] on testing process difficulties andenhancement proposals, and on our observations on indus-trial test automation [4]. In this study we further elaboratedon the test automation phenomena with a larger sample ofpolar type OUs, and more focused approach on acquiringknowledge on test process-related subjects. The surveyrevealed that test organizations use test automation only in26% of their test cases, which was considerably less thancould be expected based on the literature. However, testautomation tools were the third most common category oftest-related tools, commonly intended to implement unitand regression testing. The results indicate that adopting testautomation in software organization is a demanding effort.The lack of existing software repertoire, unclear objectivesfor overall development and demands for resource allocationboth for design and upkeep create a large threshold toovercome.

Test automation was most commonly used for qualitycontrol and quality assurance. In fact, test automation wasobserved to be best suited to such tasks, where the purposewas to secure working features, such as check moduleinterfaces for backwards compatibility. However, the highimplementation and maintenance requirements were con-sidered the most important issues hindering test automationdevelopment, limiting the application of test automationin most OUs. Furthermore, the limited availability of testautomation tools and the level of commitment required todevelop a suitable automation infrastructure caused addi-tional expenses. Due to the high maintenance requirementsand low return on investments in small-scale application,some organizations had actually discarded their automationsystems or decided not to implement test automation. Thelack of a common strategy for applying automation was alsoevident in many interviewed OUs. Automation applications

varied even within the organization, as was observablein the differences when comparing results from differentstakeholders. In addition, the development strategies werevague and lacked actual objectives. These observations canalso indicate communication gaps [58] between stakeholdersof the overall testing strategy, especially between developersand testers.

The data also suggestedthat the OUs that had successfullyimplemented test automation infrastructure to cover theentire organization seemed to have difficulties in creatinga continuance plan for their test automation development.After the adoption phases were over, there was an ambiguityabout how to continue, even if the organization had decided


15/18


to further develop their test automation infrastructure.The overall objectives were usually clear and obviouscostsavings and better test coveragebut in practise there wereonly few actual development ideas and novel concepts. Inthe case organizations this was observed in the vaguenessof the development plans: only one of the five OUs which

used automation as a part of their normal test processes haddevelopment plans beyond the general will to increase theapplication.

The survey established that 61% of the software compa-nies followed some form of a systematic process or methodin testing, with an additional 13% using some establishedprocedures or measurements to follow the process efficiency.The main source of software quality was considered toreside in the development process, with testing having muchsmaller impact in the product outcome. In retrospect of thetest levels introduced in the ISO/IEC29119 standard, thereseems to be no one particular level of the testing whichshould be the research and development interest for bestresult enhancements. However, the results from the self-assessment of the test phases indicate that low-level testingcould have more potential for testing process development.

Based on these notions, the research and develop-ment should focus on uniform test process enhancements,such as applying a new testing approach and creating anorganization-wide strategy for test automation. Anotherfocus area should be the development of better tools tosupport test organizations and test processes in the low-level test phases such as unit or integration testing. As forautomation, one tool project could be the developmentof a customizable test environment with a common coreand with an objective to introduce less resource-intensive,transferable and customizable test cases for regression andmodule testing.

Appendix

Case Descriptions

Case A (Manufacturing execution system (MES) producerand electronics manufacturer). Case A produces softwareas a service (SaaS) for their product. The company is asmall-sized, nationally operating company that has mainlyindustrial customers. Their software process is a plan-driven cyclic process, where the testing is embedded to thedevelopment itself, having only little amount of dedicated

resources. This organization unit applied test automationas a user interface and regression testing tool, using it forproduct quality control. Test automation was seen as a partof the normal test strategy, universally used in all softwareprojects. The development plan for automation was togenerally increase the application, although the complexityof the software- and module architecture was consideredmajor obstacle on the automation process.

Case B (Internet service developer and consultant). Case Borganization offers two types of services; development ofInternet service portals for the customers like communitiesand public sector, and consultation in the Internet service

business domain. The origination company is small andoperates on a national level. Their main resource on the testautomation is in the performance testing as a quality controltool, although addition of GUI test automation has also beenproposed. The automated tests are part of the normal testprocess, and the overall development plan was to increase the

automation levels especially to the GUI test cases. However,this development has been hindered by the cost of designingand developing test automation architecture.

Case C (Logistics software developer). Case C organizationfocuses on creating software and services for their origincompany and its customers. This organization unit is apart of a large-sized, nationally operating company withlarge, highly distributed network and several clients. Thetest automation is widely used in several testing phases likefunctionality testing, regression testing and document gen-eration automation. These investments are used for qualitycontrol to ensure the software usability and correctness.

Although the OU is still aiming for larger test automationinfrastructure, the large amount of related systems andconstant changes within the inter-module communicationsis causing difficulties in development and maintenance of thenew automation cases.

Case D (ICT consultant). Case D organization is a small,regional software consultant company, whose customersmainly compose of small business companies and the publicsector. Their organization does some software developmentprojects, in which the company develops services and ICTproducts for their customers. The test automation comesmainly trough this channel, as the test automation is mainly

used as a conformation test tool for the third party modules.This also restricts the amount of test automation to theprojects, in which these modules are used. The companycurrently does not have development plans for the testautomation as it is considered unfeasible investment for theOU this size, but they do invest on the upkeep of the existingtools as they have usage as a quality control tool for theacquired outsider modules.

Case E (Safety and logistics system developer). Case Eorganization is a software system developer for safety andlogistics systems. Their products have high amount of safetycritical features and have several interfaces on which tocommunicate with. The test automation is used as a major

quality assurance component, as the service stress tests areautomated to a large degree. Therefore the test automation isalso a central part of the testing strategy, and each projecthas defined set of automation cases. The organization isaiming to increase the amount of test automation andsimultaneously develop new test cases and automationapplications for the testing process. The main obstacle forthis development has so far been the costs of creating newautomation tools and extending the existing automationapplication areas.

Case F (Naval software system developer). The Case Forganization unit is responsible for developing and testing


16/18


naval service software systems. Their product is based ona common core, and has considerable requirements forcompatibility with the legacy systems. This OU has triedtest automation on several cases with application areassuch as unit- and module testing, but has recently scaleddown test automation for only support aspects such as

the documentation automation. This decision was basedon the resource requirements for developing and especiallymaintaining the automation system, and because the manualtesting was in this context considered much more efficient asthere were too much ambiguity in the automation-based testresults.

Case G (Financial software developer). Case G is a partof a large financial organization, which operates nationallybut has several internationally connected services due totheir business domain. Their software projects are alwaysaimed as a service portal for their own products, and haveto pass considerable verification and validation tests before

being introduced to the public. Because of this, the caseorganization has sizable test department when compared toother case companies in this study, and follows rigorous testprocess plan in all of their projects. The test automation isused in the regression tests as a quality assurance tool foruser interfaces and interface events, and therefore embeddedto the testing strategy as a normal testing environment.The development plans for the test automation is aimedto generally increase the amount of test cases, but eventhe existing test automation infrastructure is consideredexpensive to upkeep and maintain.

Case H (Manufacturing execution system (MES) producerand logistics service system provider). Case H organizationis a medium-sized company, whose software development isa component for the company product. Case organizationproducts are used in logistics service systems, usually work-ing as a part of automated processes. The case organizationapplies automated testing as a module interface testing tool,applying it as a quality control tool in the test strategy.The test automation infrastructure relies on the in-house-developed testing suite, which enables organization to usethe test automation to run daily tests to validate moduleconformance. Their approach on the test automation hasbeen seen as a positive enabler, and the general trend istowards increasing automation cases. The main test automa-tion disability is considered to be that the quality control

aspect is not visible when working correctly and therefore theeffect of test automation may be underestimated in the widerorganization.

Case I(Small and medium-sized enterprise (SME) businessand agriculture ICT-service provider). The case I organi-zation is a small, nationally operating software companywhich operates on multiple business domain. Their customerbase is heterogeneous, varying from finances to the agri-culture and government services. The company is currentlynot utilizing test automation in their test process, butthey have development plans for designing quality controlautomation. For this development they have had some

individual proof-of-concept tools, but currently the overalltesting resources limit the application process.

Case J (Modeling software developer). Case J organizationdevelops software products for civil engineering and archi-tectural design. Their software process is largely plan-driven

with rigorous verification and validation processes in thelatter parts of an individual project. Even though the caseorganization itself has not implemented test automation, onthe corporate level there are some pilot projects where regres-sion tests have been automated. These proof-of-concept-tools have been introduced to the case OU and there areintentions to apply them in the future, but there has sofar been no incentive for adoption of the automation tools,delaying the application process.

Case K(ICT developer and consultant). Case K organizationis a large, international software company which offerssoftware products for several business domains and gov-ernment services. Case organization has previously piloted

test automation, but decided against adopting the system asit was considered too expensive and resource-intensive tomaintain when compared to the manual testing. However,some of these tools still exist, use

software test in practice

Documents