Automatic Regression TestingIntroducing effective testing of graphical user interfaces

Oscar LundChristian Stromqvist

Department of Communication SystemsLund University

Advisor: Carina Andersson

January 22, 2007

Printed in SwedenE-huset, Lund, 2007

Abstract

Automated regression testing of software has become a more and more important part of softwaredevelopment. As the competition increases and the size and complexity of software systemsgrows, higher demands are put on the quality of the test process. The earlier defects can bedetected and attended to, the less expensive it becomes for the organization. Today, manycompanies recruit consultants for manual testing. Introducing well chosen automated tests maygreatly reduce development costs and time as well as ensuring a high quality of the product.

This report describes different methods to use when introducing automated testing withprimary focus on graphical user interfaces. Different test reference models has been reviewedand a number of test tools as well as scripting techniques has been evaluated. Although thissubject has been treated in numerous other studies, the focus often lies on the theory anddifficulties of graphical user interface testing. It is rare to find actual examples of how theseautomated tests are implemented. Since both testing tools and methods evolves rapidly, thereports are often outdated and not applicable on the software today. Finally, a test strategy anda test environment which suits the needs and resources of Beijer Electronics AB are presented.

In conclusion, Test Process Improvement model is the reference model best suited for BeijerElectronics AB. Furthermore, TestComplete is the test tool recommended and Keyword drivenscripting is the choice of scripting technique.

i

ii

Preface

This master thesis is the result of a project work at the company Beijer Electronics ProductsAB, located in Malmo, in cooperation with the Department of Communication Systems at LundUniversity. It has been 20 weeks of hard labour, but also very interesting, educational and fun.We would like to take this opportunity to thank several people who have had a very positiveeffect on this Master Thesis.

• Daniel Bjorkkvist, our supervisor at Beijer Electronics Products AB. A very patient manand a good mentor.

• Ola Andersson, our supervisor for the last couple of weeks at Beijer Electronics ProductsAB.

• Per Runeson, our supervisor during the first three weeks at the Department of Commu-nication Systems.

• Carina Andersson, our supervisor at the Department of Communication Systems.

• Tommy, the chef at Beijer’s restaurant for delicious food every single day.

• All testers and developers at Beijer Electronics AB.

Malmo, January 22, 2007

Oscar Lund Christian Stromqvist

iii

iv

Table of Contents

1 Introduction 11.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.4 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.5 Intended audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.6 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.7 Outline of this report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2 Survey 72.1 Exploring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.2 Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.3.1 Questionnaire . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.3.2 Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3 Reference models 173.1 TMM - Test Maturity Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173.2 TPI - Test Process Improvement . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.2.1 Key Areas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.3 MTPF - Minimal Test Process Framework . . . . . . . . . . . . . . . . . . . . . . 213.3.1 Responsibility Areas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4 Test tool 254.1 TestComplete 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284.2 TestPartner 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294.3 AutoMate 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304.4 QuickTest 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

5 Scripting Techniques 335.1 Linear scripting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

v

5.2 Structured scripting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345.3 Shared scripting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355.4 Data Driven scripting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355.5 Keyword Driven scripting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

6 Suggestion 396.1 Reference model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396.2 Test tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 406.3 Scripting Technique . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

7 Implementation 437.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

7.1.1 Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

7.1.2 Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

7.1.3 Supporting Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

7.1.4 Projects Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

7.1.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

7.2 Test Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 487.3 Controller Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 507.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

8 Conclusion 538.1 Proposition evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 548.2 Prototype evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 548.3 Future improvements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

References 57

A Questionnaire 59A.1 Organization Stage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59A.2 Tool Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61A.3 Tool Features Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62A.4 Identifying Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63A.5 Programming language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64A.6 Test/Trouble Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65A.7 Final Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

B Interview Questions 67B.1 Organization Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67B.2 Maturity identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

C Implementation 71C.1 Controller script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71C.2 Supporting scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74C.3 Project Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

vi

Chapter 1Introduction

1.1 Background

Beijer Electronics AB is, like many other small but growing companies, in a phase where thenext natural step in testing and development is the introduction of automatic regression testing.As of today, there exists a limited number of reports and studies regarding automatic regressiontesting with focus on graphical user interfaces. Furthermore, these are often outdated, i.e. notapplicable on today’s software technologies. Others are too theoretical and therefore sometimesvery difficult to apply in practice.

In the present, Beijer Electronics AB has no automatic testing at system and acceptancelevel. The company rely on manual testing to ensure the quality of their products. Manualtesting is currently performed in the end of a product’s development process and do not receiveas much time as it should, sometimes due to delays in the development. The goal of introducingautomatic testing is to be able to perform tests as early as possible in the software developmentprocess.

Beijer Electronics AB produces builds of their software on a daily basis. By testing thesebuilds with some selected automatic regression tests every time, time is saved since certain defectsare detected immediately. In this way and by having most regression tests run automatically,the organization ensures a higher quality of their product.

Beijer Electronics AB develops and produces human machine interface products, i.e. ter-minals controlling industrial robots and machines. The terminals communicates directly withPLCs integrated with the machines. These graphical terminals is delivered to customers withoutany PLC controlling software installed. Instead the terminals are delivered with an applicationthat allows the end user to build their own graphical user interface to control the PLC. At BeijerElectronics AB, this application is referred to as Neo.

Neo is a design tool, similar top Visual Studio developed by Microsoft [?]. It is developedwith Microsoft .NET 3.0 [?]. This is a new framework which very few test tools have supportfor. Neo offers the possibility to design and implement an application with buttons, text fields,meters etc. The user can then compile the application and transfer it to the terminal. In thisway, a client designs a graphical user interface suiting the needs for different industrial machinesor robots. A screenshot of Neo can be found in Figure 1.1.

1

2 Introduction

Figure 1.1: Screenshot of Neo build 920

To ensure the quality of Neo, Beijer Electronics AB is interested in introducing automaticregression testing as an additional quality assurance to the test process. This study takes placein parallel to the development of Neo and is executed with continuous feedback from testers anddevelopers. The idea is that the developers will be of assistance during the implementation ofthe test environment. For a deeper description of how this assistance works in practice, refer toSection 7.2.

A very important reason for choosing automatic regression testing over manual testing isthat automatic regression testing offers the possibility to test a certain functionality in the exactsame way over and over again. This helps in recreating an error and thereby identifying whythe error occurs as well as ensuring if a defect has been attended or not. This is hard to do bymanual testing since it is impossible to repeatedly perform a test case exactly in the same way.

Introduction 3

1.2 Objective

The main objective of this report is to offer a both theoretical and practical guide on how tointroduce automatic regression testing in small size software companies. The concept of thisguide has been that it should be easy to understand and be relatively manageable to apply andimplement. This study presents a strategy on how an organization can evaluate and introducetest strategies, test tools and test structures suiting their specific needs. As a case study, Neo,developed at Beijer Electronics AB has been used.

1.3 Approach

The first step is to identify and formulate the problem. The next step is to acquire the basicknowledge needed to attack the problem in an effective way. This research incudes identifyingseveral important areas, such as the choice of reference model, test tool and scripting techniques.

To motivate the choices made when selecting reference model and test tool, a precedingresearch is made. By handing out forms to employees as well as having interviews and discussionswith group leaders, the understanding for the company is increased and helps in giving a morewell-founded proposition fitting the company’s needs and goals. The results of the questionnaireis intended to give a basic understanding of the level of knowledge and experience of employees,as well as the expectations of the test tool being chosen.

The evaluations of reference models, test tools and scripting techniques are made solely bythe authors of this report. However, considerations form the results of the questionnaire and theinterview are taken into account.

Figure 1.2 shows the work procedure used to acquire the information needed to make awell-grounded decision. The initial proposition presented is by no means final. As feedback isreceived, the proposition is reviewed and modified. This is repeated continuously until bothsides mutually agrees to a final version of the test strategy.

As the final proposition is formed, the support state is entered and a test strategy is definedand presented to the company and all personal concerned.

The final steps are implementation and evaluation. As the arrows in Figure 1.2 applies, thereis an open and continuous dialog with all parts concerned throughout the entire process of theintroduction of automatic regression testing.

4 Introduction

Explore

Reference Model

Suggestion

Support

Implementation

Evaluation

SurveyInterview

Test Process ImprovementTestabilty Maturity ModelTest Maturity ApproachMinimal Test Practise Framework

FeedbackReview

Figure 1.2: Work flow

1.4 Scope

This report is not to be seen as a complete reference to automatic regression testing, but ratheras a guide when introducing automatic regression testing in a test environment of a small ormedium-sized organization. To get a deeper understanding for certain areas like reference modelsand test structures, it is recommended to study the literature in Chapter References. The choicesmade in this report focus on Beijer Electronics AB and may not be the right solution for anotherorganization.

The implementation of the test environment is to be seen as a prototype. The number oftest cases created is very limited and basic, but are intended to function as a guideline for theimplementation of other and more advanced test cases.

1.5 Intended audience

This report is intended for higher education students with focus on software development andbasic knowledge in software testing. This study also is of interest for smaller or medium-sizedcompanies aiming to introduce automatic testing in their test environment.

Introduction 5

1.6 Glossary

Subject Explanation

TMM Test Maturity Model

TMap Test Management approach

TPI Test Process Improvement

MTPF Minimal Test Practice Framework

Regression Testing Retesting of a previously tested program, to ensure that modificationshas not caused new defects.

Table 1.1: Glossary

1.7 Outline of this report

The first part of this report, Chapter 2 Survey, describes the exploring part of this master thesis.This is used to get an understanding for the organization as well as formulating and identifyingthe problem.

The second part, Chapters 3, 4 and 5, evaluates reference models, test tools and scriptingtechniques respectively.

The third part, Chapter 6 Suggestion, presents the results of the evaluations. Here are thechoices made and motivated considering reference model, test tool and scripting techniques.

The forth part, Chapter 7 Implementation, illustrates the actual implementation of the teststructure. This is the result of all preceding chapters and ideas.

The final part, Chapter 8 Conclusion, evaluates the prototype from both an own perspectiveand tester feedback view. This chapter also includes future improvements.

6 Introduction

Chapter 2Survey

The first natural step when introducing automatic regression testing is not only to identify areference model suiting the organization’s current level of maturity of the test process, but alsoto setup a test tool fitting the company’s needs with cost, usability, functionality etc. taken intoconsideration.

To be able to make these decisions, an extensive analysis of the organization is needed. Withwell-grounded and relevant questions to all employees, concerned with the test process, the ideais to create a test strategy and test environment, suitable for the organization. In this chapterthe process that lies as a foundation to the proposition is described. If the management approvesof this suggestion, it is used as a starting point of how the company will work with testing, testimprovements and test planning.

In Section 2.1 Exploring, the process of gathering the information needed to get a clearimage of what the organization expects and requires, is described. In section 2.2 Interview,recommended questions for test management for understanding the level of maturity of theorganization, is discussed.

2.1 Exploring

To achieve a satisfying and a well functional test strategy fitting the organization it is of greatimportance to get a deep understanding for the activity. By having insight in the organizationalstructure, how the testers are working and what knowledge and experience they have as well astheir expectations of the test process, a better and more well-founded decision regarding teststrategy can be made. It is also important that the test strategy with minimum efforts can beintegrated in the current organizational structure. The advantage of this thorough organizationalunderstanding is minimizing the risk of time consuming and resource absorbing reconstructions.As a positive side-effect a greater trust is built among the employees, mainly because the teststrategy presented is well-founded and somewhat familiar.

A first step is collecting information about the activity. This is done preferably with aquestionnaire sent to all employees concerned with the test process, for example software qualityassurance managers, testers and developers. The main purpose with the questionnaire is to getan understanding of what test personal expects and demands of the test tool they eventuallywill be working with. Furthermore it is important to get an insight on how the test environmentis structured initially, regarding experience, knowledge and work assignments of the employees.[1]

7

8 Survey

The questionnaire is found in Appendix A. The first section of the questionnaire treatsorganizational questions with focus on the individual tester’s experiences and views on thecorporation. These questions do not lie as a foundation when choosing reference model, butrather gives an indication on what level a test process can be established. The questions ismainly fetched from Burnstein [1], but are reworked to be relevant for smaller companies likeBeijer Electronics AB.

The second and third section of the questionnaire treats the needed functionality and featuresof the test tool. The answers are the very foundation for the choice of test automation tool. Athorough evaluation of different test tools is made, where various functionalities and features israted, see Chapter 4. These questions gives an idea of what properties of the test tools that is ofhighest importance for the testers. Examples of areas the questions handles are support, cost,stability and functionality. In chapter 4 the results are evaluated and in Section 6.2 is explainedin detail how they have affected the choice of test automation tool. The questions in chapter1 and 2 in the questionnaire are collected from Burnstein [1] and the questions in chapter 3are from Fewster [2]. The remaining questions is the results of our own experience in softwaretesting and to some extent feedback from software testers at Beijer Electronics AB.

With a test automation tool one can create many different types of tests. This often requiressome programming skills regarding the construction of the tests. This is treated by the questionsin chapter 5 in the questionnaire, as what programming languages the testers have experiencefrom and prefer. This, of course, also inflict on the choice of test tool.

In the end of the questionnaire there is a chapter where the concerned employees havethe opportunity to leave comments regarding test and defect reporting. These comments are afactor when choosing a reference model, since it is important to know how everything works in thepresent and if there is a possibility to make improvements. Finally there is a final comments fieldwhere the readers can add thoughts on the proposition and if they feel that there is somethingleft out.

By taking the feedback from the testers into consideration and letting them affect the out-come of the new test strategy, it is easier to carry out all major changes since all employees feelthat they are part of the new solution. Hopefully this leads to that employees feel a responsibilityto make sure the new reorganization really works.

2.2 Interview

The questionnaire gives an understanding of the skills and the expectations of the employees atthe test department. To get a deeper understanding of the organization, which level of maturityit is in, as well as what the management expects, an open discussion with the test managers isrequired.

The discussion is based on several interview questions which can be found in AppendixB. These questions is primarily based on suggested interview questions from Burnstein [1].Questions irrelevant for Beijer Electronics AB has been removed. The results will underlie thetest strategy which is to be introduced in the company. An analysis of the answers can be foundin section 6.1, as well as a deeper explanation on how they have affected the choice of referencemodel.

The first part of the interview, Organizational background, treats more practical questions,such as number of full-time and part-time employees. This is of interest when deciding a reference

Survey 9

model fitting the organization’s size.Second and last part of the interview mainly treats questions that will be a foundation when

identifying which level of maturity the company has achieved. The questions has been fetchedfrom a recommended template, developed by Burnstein [1] for reference model TMM. However,MTPF, TMM and TPI all include the process of identifying the maturity of the organization ortest process. In this study, all interview questions are selected to be relevant for all three testframeworks. This part is divided into three smaller parts. The guideline for the questions isthat if more than half of the answers in each part are positive, than that part’s level of maturityis achieved. If more than half of all parts has reached maturity, then the total maturity level isconsidered achieved. [1]

First part treats Testing Goals and Policies. Next part focuses on Test Planning Process.Final and last part treats Testing Techniques and Methods.

2.3 Results

2.3.1 Questionnaire

The questions handed out to the seven employees were answered and summarized in tables. Themajority of all testers, including all test managers, have been involved in this research. From thedevelopment department, two senior software architects were selected. These seven employeeswere considered to have the knowledge and insight to represent the company. The reason forselecting more testers than developers is that the test department should have more to say inthis matter.

To get a picture of the organizational structure the first two questions focused on the individ-ual work position of the employees, as seen in Table 2.1 and Table 2.2. Each cross represent ananswer from an employee, while the last column represent the average. As Table 2.1 illustrates,approximately one third of those asked are developers and the rest is part of the test department.

Which best describes your current position?Software Quality Assurance Group Leader x x 29%

Software Quality Assurance Group Member x x x 43%

Software Engineer (developer) x x 29%

Table 2.1: Employee position

Table 2.2 does not only show the responsibilities of the employees, but also hints of whichresponsibility areas within the test process that are covered.

10 Survey

What is your current test-releated duties?Test policy and goal development x 14%

Test planning x x 29%

Test case and test procedure design x x 29%

Test execution x x x x 57%

Maintenance of case database 0%

Standards development 0%

Reviews and audits 0%

Status tracking 0%

Test training 0%

Hiring and recruiting x 14%

User/Client communications x 14%

Test process control x 14%

Test process evaluation x x 29%

Test process improvment x x 29%

Defect prevention 0%

Usability testing x x 29%

Tool evalutaion x x 29%

Table 2.2: Employee responsibilities

When evaluating different test strategies it is important to know what knowledge the testershave when it comes to different reference models. As the Table 2.3 implies, the Test ProcessImprovement Model, TPI is most widely known among the employees. TPI is also currentlyused as a reference model at Beijer Electronics AB.

Are you familiar with any of the following test practise frameworks.TMM (Test Maturity Model) x x 29%

TPI (Test Process Improvment model) x x x 43%

MTPF (Minimal Test Practice Framework) 0%

TMap (Test Management approach) x x 29%

Table 2.3: Reference models

The next question treats the amount of work experience there exists among the employees.The Table 2.4 indicates that the testers of Beijer Electronics AB are well trained and experienced.Therefore no major difficulties is expected when introducing a more complex test environment.The basics of testing has also been left out of the proposition and this study, since it is consideredthat the target group already has acquired this knowledge.

Survey 11

What is the extent of your experience in software testing?Less then 1 year 0%

1 to 2 years x x 29%

2 to 3 years 0%

More than 3 years x x x x x 71%

Table 2.4: Work experience

The next question, as seen in Table 2.5, gives an idea of how structured the test processis in the present. This gives a hint of difficult it can be to introduce a highly structured testenvironment, compared to the current structured level. As seen in Table 2.5 most of the em-ployees consider the test environment to be somewhat structured, which helps when introducingautomatic regression testing.

How would you generally characterize the nature of your testing process?Ad Hoc 0%

Informal x 14%

Somewhat structured x x x x x 71%

Highly structured x 14%

Table 2.5: Level of structure

When evaluating test automation tool it is important to know how the testers prioritizedifferent properties of the tool. As Table 2.6 shows, the most prioritized properties are Ease ofUse and Functionality. Later on, when evaluating test tools, see Chapter 4, the average scoreof each property is used as a weight. For a description of each property, refer to Table 4.2.The Functionality category consists of several different tool features as described in Table 2.7.The employees rated the importance of each property on a scale from 1 (the lowest) to 7 (thehighest).

Tool Evaluation Ease of use 7 7 6 7 5 5 6 6,1Power 3 6 7 5 6 6 7 5,7Robustness 5 6 6 6 6 4 6 5,6Functionality 4 6 6 7 7 7 6 6,1Ease of insertion 6 4 5 4 5 6 2 4,6Quality of Support 4 6 5 1 4 5 5 4,3Cost 3 5 3 3 1 5 3 3,3

Table 2.6: Tool property evaluation

As seen in Table 2.7, the most valued features are GUI testing and Script support, which wasexpected. The employees rated each feature on a scale from 1 (the lowest) to 5 (the highest).

12 Survey

Tool Features EvaluationRemote handlig 3 1 3 3 2 1 2 2,1Recording 4 5 4 4 2 5 5 4,1Native .NET testing 3 3 5 5 4 4GUI Testing 5 4 5 4 5 5 5 4,7Script support 4 3 5 4 4 5 5 4,3Plugin support 4 2 2 4 4 4 5 3,6API support 3 3 5 4 3 3,6COM support 3 4 5 5 4 3 4Distribuation of test reports 3 2 4 3 5 3 5 3,6Compare test results 5 3 4 4 3 4 2 3,6Other

Table 2.7: Tool functionality evaluation

The next question treats specific test areas for the product (Neo) being developed at BeijerElectronics AB. It is not always the case that these areas are monitored together with automaticregression testing. However, it is of interest to find out if there is a need to cover these test areasin the test environment. As seen in Table 2.8 memory leaks has a high priority to be tested.The employees rated each area on a scale from 1 (the lowest) to 5 (the highest).

Identifying Test CasesExecution time of NEO 4 4 3 2 4 3 2 3,1Execution time of generated binaries 2 3 3 4 3Response time of NEO 4 4 3 3 4 4 3 3,6Response time of generated binaries 2 3 3 4 3Memory usage of NEO 4 5 4 4 3 4 2 3,7Size and memory usage of generated binaries 4 5 3 4 4Memory leaks in NEO 4 5 5 5 2 4 4 4,1Memory leaks in generated binaries 5 5 3 5 4,5

Table 2.8: Additional test areas

The last two questions determines the level of experience and skills testers have in differentscripting languages. This can be of value when choosing and evaluating test automation tool,depending on how test cases are created, for example by recording or writing test scripts. Asseen in Table 2.9 and 2.10 the testers have the most knowledge and skills in Visual Basic script,which can be worth noticing when evaluating test automation tools. The employees rated theirexperience and knowledge in each programming language on a scale from 1 to 5, where 1 is noknowledge and 5 is experienced.

Survey 13

Programming language (prefer)VBScript x x x x 57%JScript 0%C#Script x x 29%DelphiScript x 14%

Table 2.9: Preferred programming language

Programming language (skills)Visual Basic 2 2 3 3 2 4 2,7Java 2 1 1 1 2 1,4C# 2 3 1 1 1 5 2,2Delphi 3 1 1 1 5 2,2

Table 2.10: Programming skills

2.3.2 Interview

The interview took place approximately five weeks from project start. Fredrik Persson, SoftwareQuality Assurance Manager, was the objective for the interview. The interview is intended to givean understanding of the organizational structure and current maturity level. This helps whendeciding an appropriate reference model and to what level of difficulty the test environmentcould be built. All questions in this interview were gathered from Burnstein [1], with somequestions slightly modified. The number of questions are also decreased, so that only the mostrelevant questions, suitable for a smaller company, are used.

The first section of questions according to Table 2.11 treats size and structure of the organi-zation. According to Burnstein [1] these questions are important when identifying the maturitylevel and deciding possibilities of improvements. As seen in Table 2.11, the human resources intesting of Beijer Electronics AB is rather limited. It is therefore important to develop a testsystem which do not requires to much costs and resources in maintenance. By introducing au-tomatic regression testing a large portion of the time caused by manual testing can be avoidedand thereby allowing testers to concentrate on less tedious tasks. Accordingly to Table 2.11,compared to software development, a lot of resources are put into testing. However, it is impor-tant to emphasize that testers not only works with software testing, but also performs hardwareand driver testing.

14 Survey

How many people are employed in the organization being assessed?Total employees 70Number engaged in software development 12Number engaged in software testing 7

Please describe the percent of staff engaged in testing as follows?Fulltime 43%Part time 14%Consultants 43%

Are the responsibilities for test process improvement clearly defined and supported?Very well defined

How is the testing group organized?Developers do the testing No

Test group within development report to project manager No

Separate test group, report to test manager YesPart of Software Quality Assurance group Yes

Table 2.11: Organizational background

Next three sections focuses on identifying the maturity level of the company. Questions seenin Table 2.12 focus on testing and debugging goals. It can be stated that the testing goals andpolicies are either well defined or under development. This means that the organizational basisfor a functional test environment has been established.

Develop testing and debugging goals and policiesHas a committee on testing and debugging been established? Yes

Have policies, goals, activities, and tools for the testing processbeen identified,documented and approved? Under development

Do the software testers follow a written organizational policy for testing when test planning? Under development

Are basic measurements used to determine achievement of testing goals? Yes

Have the testing policies and goals been developed with inputs from user/client groups with respect to their needs? Yes

Do developers/testers log defects into the repository on a consistent basis? Yes

Are testing/debugging policies and goals periodically reviewed? Yes

Table 2.12: Maturity identification, part 1

Next section is concentrated on test planning. According to Table 2.13 there is room forsome improvements in the area of test planning.

Survey 15

Initiate a test planning processHas an organizationwide test planning commitee or group been established? No

Are there procedures for test planning? Yes

Is there adequate support and funding for test planning for all projects? Yes

Are test goals/objectives used as a basis for test planning? No

Have test plan templates been developed? (for earlier projects?) Yes

Are there appropriate planning tools available for test planning? Yes

Are test managers trained properly to develop test specifications, test designs and test cases? Yes

Are testers trained properly to develop test specifications, test designs and test cases? No

Are test-related risks considered when developing test plans? Yes

Are estimates (time, budget, tools) available from past projects for use in test planning? No

Is test planning done at the acceptance level? No

incident reports, and test summary reports defined in organizational documents? Yes

Are other test-related items such as test transmittal reports, test logs, test incident reports, and test summary reports completed for each project? Yes

Do testers collect and store basic test-related measurements? No

Are the basic test measurements used to ensure that basic testing goals have been met? Yes

Table 2.13: Maturity identification, part 2

Final section of the maturity identification questions evaluates the current situation regardingdifferent test techniques and methods. As Table 2.14 shows, quite a few improvements are neededhere also.

16 Survey

Institutionalize basic testing teqniques and methodsHas a committee or group been formed to evaluate and recommend a set of basic testing techniques, methods, and tools? Yes

Have the recommendations of the group been documented, distributed, and approved? No

Have basic tools and techniques been included in test policies? No

Have appropriate forms and templates been designed to support basic testing techniques? No

Are adequate resources provided by upper management to support the use of basic testing techniques and methods, as well as basic testing tools? Yes

Have testers been trained to apply the basic tools, forms, and methods? Yes

Are basic testing techniques, and methods applied on an organizationwide basis? No

Are basic testing techiques and methods reviewed periodically? Yes

Is testing planned and implemented at multiple levels (unit, integration, system, etc.)? No

Table 2.14: Maturity identification, part 3

From the results of these questions a reference model is chosen and the maturity level isidentified. This is presented in Section 6.1.

Chapter 3Reference models

This chapter gives an overview of a selection of widely used reference models. The choice ofreference model describes how the company or organization will work with the test environment.In this chapter, a number of different test improvement models are treated. The focus in thisstudy is on the models TMM, TPI and MTPF.

All test improvement models strives for a more structured and effective test process andorganization. According to Pol, the goal of a test improvement model is a test process methodcalled TMap. For further reading on TMap, refer to Pol [3]

3.1 TMM - Test Maturity Model

TMM or Test Maturity Model is a test strategy built upon five levels of maturity. When atest process has achieved a number of specified demands on one level, the test process can beconsidered being on that level. [1]

An organization that does not have any testing or a highly unstructured test environmentmainly based on ad hoc is in the initial state at level 1. For a company to climb to the next levelit is required to introduce a more structured testing. This involves different test techniques, likeWhite-Box and Black-Box testing, but also development of test- and debugging goals as well asa test planning process.

To climb further up in the level hierarchy additional measures are needed. In term this meansthat testing no longer is a part of the software development process, but rather an independentprocess starting as early as the software development itself. This means introducing different testactivities at various states in the software development and having skilled groups established,each responsible for different parts of the test process, such as planning, execution and evaluation.

17

18 Reference models

Level 1: Inital

Level 2: Phase Definition

Institutionalize basic testing techniques and methods .Initiate a test planning process .Develop testing and debugging goals .

Level 4: Management and Measurement

Software quality evalutaion .Establish a test measurement program .Establish an organizationwide review program .

Level 3: Integration

Control and monitor the testing process .Integrate testing into the software life cycle .Establish a technical traning program .Establish a software test organization .

Level 5: Optimization/Defect Preventionand Quality Control

Test process optimization .Quality control .Aplication of process data for defect prevention .

Figure 3.1: TMM maturity level structure

A smaller company or an organization with limited resources allocated for testing, may havetrouble reaching level 2 and even more difficulties reaching higher levels. The magnitude of thesesteps not only makes it difficult and demanding but also less motivating.

3.2 TPI - Test Process Improvement

Test Process Improvement, TPI, is a test practise framework developed with the test methodTMap [3] in mind. The main purpose is to allow a more balanced test improvement strategy.This means that companies continuously works with the improvements and can follow the levelsof maturity through a so called test maturity matrix [4], see Figure 3.2

Reference models 19

Key Area Scale 0 1 2 3 4 5 6 7 8 9 10 11 12 13

Test strategy A B C D

Lifecycle model A B

Moment of involvement A B C D

Estimating and planning A B

Test-specification techniques A B

Static test techniques A B

Metrics A B C D

Test automation A B C

Test environment A B C

Office environment A

Commitment and motivation A B C

Test functions and training A B C

Scope of methodology A B C

Comunication A B C

Reporting A B C D

Defect management A B C

Testware management A B C D

Test process management A B C

Evaulation A B

Low-level testing A B C

Controlled Efficient Optimizing

Figure 3.2: TPI Test maturity matrix

The matrix consists of twenty different key areas, each representing areas all open for con-tinuous improvements. Every area has a number of maturity levels. The grading goes from Ato D, where A is the lowest level and D is the highest level. Each maturity level is placed on ascale from 1 to 13, illustrating the significance of fulfilling every individual level. The space inbetween the different levels describes the relation to other key areas. It is a wise choice to havedifferent levels of maturity evenly distributed over the matrix. For example, it is not particularyeffective to have a high level of evaluation without a well defined test strategy.

3.2.1 Key Areas

This section gives a short summary of all the key areas in the TPI maturity matrix, see Figure3.2. For further explanations, refer to [4].

Test strategyThis area focuses on how well defined the test strategy is on different levels of testing. It isimportant to detect defects in a stage as early as possible.

Life-cycle modelA test process consists of different phases, such as planning, preparation and execution. Eachphase contains predefined activities. Worth mentioning are purpose, techniques and tools. Whenusing a life-cycle model one gets a much better overview of the test process and thereby can plan

20 Reference models

and execute it in a more efficient way.

Moment of involvementDefines when the test process should start in relation to the software development. The testprocess consists of more than just the test execution.

Estimating and planningDefines when different activities within the test process is to be carried out and what necessaryresources that should be available.

Test-specification techniquesDescribes how different test cases are generated from various testing goals.

Static test techniquesStatic tests means that inspections of the product are made without running it, for example byreviewing the source code.

MetricsBy continuously examine the product and the test process, metrics is introduced. This gives anopportunity to measure the quality of the entire test process.

Test automationDescribes to what degree supporting tools is used to automate different parts of the test process.

Test environmentDescribes to what extent hardware, software, communication means, facilities and proceduresfulfill the needs of the testers.

Office environmentThe test staff need adequate working conditions to be as efficient as possible in test process.

Commitment and motivationWithout a committed and motivated test staff, an efficient test process can not be expected.There are several ways to increase the commitment and motivation, for example by assuringthere is adequate time and resources.

Test functions and trainingIt is important to have a set of well-defined test techniques, test methods and test functions, aswell as experienced and skilled testers with concrete roles within the test team.

Scope of methodologyFor every test process in an organization a specific test method is used. If these test methodsdiffers from one process to another, this results in a decreased efficiency. It is desirable to usea sufficient generic method that can be applied on each test process, but still contains enoughdetail to be efficient each time.

Reference models 21

CommunicationThe communication between the people involved in a test process are extremely important, notonly for creating good conditions, but also to ensure the quality of the product

ReportingTesting is mainly about quality assurance. Reporting aims at giving the client an insight in thetest process and the quality of it.

Defect managementAdministration of defects is important to be able to follow the life-cycle of a reported error andthereby analyze the quality trend in detected defects.

Testware managementVersion control systems, structuring of test plans and specifications are important parts of atestware system, to make it run as efficient as possible.

Test process managementThere are several steps and activities to be carried out in a test process, to make it well-structuredand effective. The most important are planning, execution, checking and action.

EvaluationBy continuously inspecting intermediate products, for example requirements and functional de-sign, defects can be detected at an earlier stage.

Low-level testingLow-level testing, such as unit tests and integration tests, are performed exclusively by develop-ers. This is important in helping finding defects as early in the test process as possible.

3.3 MTPF - Minimal Test Process Framework

Smaller and medium sized companies often have difficulties in applying well known and largereference models. These are in most cases intended for larger organizations with resources,development teams and test groups. For smaller expanding companies it can be difficult toapply the recommendations suggested in other test improvements models to increase in level ofmaturity. With this in mind, MTPF was developed, which can be seen as a simpler alternativeto other more known test improvement models. MTPF was developed by Karlstrom, Runesonand Norden and has been evaluated by local companies in Lund and Malmo, Sweden. [5].

MTPF is a model suited for growing companies mainly practicing Ad Hoc in their testingenvironment. As Figure 3.3 shows, the test improvements are applied as the company is growing.There is no specified time plan when advancements should be done. These are performed onlywhen the organization feel they are ready and all requirements are fulfilled.

22 Reference modelsP

hase

3 (

30+

)

Maintain & Evolve System

Define Teams Perform Inspections Risk Management

Pha

se 2

(~

20)

Create System Define GoalsPerform

WalkthroughsTest Cases

Pha

se 1

(~

10)

Define SyntaxDefine

ResponsabilityUser Checklists

Basic Administration of Test Environment

Test Plan

Cat

egor

y

Problem & Experience Reporting

Roles & Organizations

Verification & Validation

Test Administration Test Planning

Coordinate Software Quality Assurance

Figure 3.3: Overview of MTPF

3.3.1 Responsibility Areas

In this section, the different areas in the MTPF matrix, according to Figure 3.3 is described.

Problem and experience reporting

Phase 1 Define SyntaxDefine a standard and a syntax to follow when reporting defects. This mini-mizes the risk of misunderstandings.

Phase 2 Create SystemIntroduce a centralized system for defect reporting.

Phase 3 Maintain & Evolve SystemVerify that error reporting works satisfying and continuously improve themeasures taken in the earlier phases, to further raise the quality.

Table 3.1: Problem and experience reporting

Reference models 23

Roles & Organization

Phase 1 Define ResponsibilityTesting responsabilty is defined within the company.

Phase 2 Define RolesThe roles for the manager, testers and developers are assigned. Everyone hasclear responsibilities.

Phase 3 Define TeamsAn independent test team is created and separated from development.

Table 3.2: Roles & Organization

Verification & Validation

Phase 1 Use ChecklistsChecklists are used to perform the most important activities such as GUI-testing and platform testing.

Phase 2 Perform WalkthroughsWalktroughs should be done before software is ready for execution. This ispreferably done in the design phase.

Phase 3 Perform InspectionsInspections is more often used rather than walkthroughs. When an organi-zation grows, a more formal method for evaluation is required.

Table 3.3: Verification & Validation

Test Administration

Phase 1 Basic Administration of Test EnvironmentTest environment must always be available. Administration is responsible fortest environment for each and every single project, as well as documentationand updates.

Phase 2 Test CasesTest cases is developed to ensure that the most common situations and ac-tivities are handled in a correct way.

Phase 3 Risk ManagementRisk managment focuses on identifying problems and risks in the beginningof each project.

Table 3.4: Test Administration

24 Reference models

Test Planning

Phase 1 Test PlanA test plan is created to gather all test related information.

Phase 2 & 3 Coordinate Software Quality AssuranceRoutines to ensure quality is introduced. This assures that softwarehas reached a certain level of quality before release.

Table 3.5: Test Planning

3.4 Summary

In this chapter three different test process improvement models has been reviewed. As alreadyconcluded it is important to choose a reference model fitting the company’s needs. In chapter 6a reference model is chosen based on the conclusions made in this chapter. In other words, thischapter only gives a review of each test framework and in chapter 6 the models are summarizedand the most appropriate is recommended.

Chapter 4Test tool

In this chapter different test automation tools are described. A thorough review of each testtool is done in order to make a decision, of course with some requests from the employees inmind, of which test tool that will be used in the automatic regression testing environment. Allthe evaluated test applications are specialized in Graphical User Interface (GUI) testing. Asummary of the test tool evaluation is presented in Figure 4.1. TestComplete were evaluatedfirst and the ratings were set relatively to the expectations. These values were then consideredto be reference values and the remaining three tools were rated relatively to TestComplete. Thesummarized rating for each test tool is calculated by Formula (4.1), where ri is the rating for acategory and wi is the weight for that category.

S =∑

ri ∗ wi∑wi

(4.1)

25

26 Test tool

Category Test Complete Test Partner AutoMate 6 Quick Test WeightEase of use 4 3 5 2 6,1

Power 3 3 4 3 5,7

Robustness 4 3 3 4 5,6

Functionality 4,1 3,4 2,9 3,1 6,1

Ease of insertion 3 2 3 2 4,6

Quality of support 3 3 2 3 4,3

Cost 4 2 3 2 3,3

Summary 3,6 2,8 3,4 2,8

Functionality Test Complete Test Partner AutoMate 6 Quick Test WeightRemote handling 5 5 5 5 2,1

Recording 4 5 2 2 4,1

.NET 5 4 2 4 4

GUI-Testing 4 4 5 3 4,7

Script support 4 3 3 3 4,3

Plugin support 4 1 1 1 3,6

API support 4 3 3 4 3,6

COM support 4 3 3 3 4

Distribution of reports 3 4 3 4 3,6

Compare 4 2 2 3 3,6

Average 4,1 3,4 2,9 3,1

Table 4.1: Test Tool Evaluation. Scale 1 to 5

The ratings found in the category Functionality comes from the results of the functionalityevaluation, see second part of Table 4.1. The column Weight is the average of values fetchedfrom the questionnaire, i.e. how the testers rated each and every category and functionality. Anexplanation of each category can be found in Table 4.2

Test tool 27

Category Description

Ease of Use The tool has an intuitive interface, which makes it easy for an un-experienced user to be able to work with it. How well the programhandles both predictable and unpredictable errors as well as excep-tions in test scripts are evaluated. Furthermore, steepness of thelearning curve is taken into consideration.

Power Number of features, usefulness of the commands are both impor-tant factors of this category. How versatile the program is and thepossibility to validate objects and structures are other importantaspects.

Robustness The reliability and the possibility to recover from program failuresare considered. How compatible different versions of the programare are also important.

Functionality Ten specific functionality requests from the testers are evaluated.These are Remote handling, Recording, .NET support, GUI-testing,Script support, Plugin support, API support, COM support, Dis-tribution of Reports and Comparison of Files and Logs.

Ease ofInsertion

The difficulty of introducing the software in the company, i.e. howwell does it fit with the organization’s present situation. Anotherimportant aspect is if the employees already have the experienceand proper training needed to manage the test tool.

Quality ofSupport

Does the company developing the test software have a history ofgood support? That is for example, does the company release up-dated versions on regular basis? Does the company offer mail, phoneor online support? Is the software well documented and is theresupportive information to be found on other sites?

Cost License costs and other costs connected with the tool, such as train-ing, maintenance and updates.

Table 4.2: Categories in Tool Evaluation

Following sections is a summarized evaluation of the test tools. Only the most differentialand characteristic features are described in every category.

28 Test tool

4.1 TestComplete 4

TestComplete [7] is developed by Automated QA.

Ease of UseTestComplete has a very intuitive user interface. It is easy to create and maintain test cases.The basic features is very easy to learn by just using the recording function, which translatesmouse movements and clicks to various scripting languages. The documentation and samplesare very useful to get familiar with the program. However, when it comes to the more complexfeatures of TestComplete it is difficult to find relevant information.

PowerTestComplete is an extremely powerful testing tool. Due to its .NET and scripting support ithas very few limitations. Currently there is only a Beta version released of TestComplete version5. It is very unstable, however it offers full support for .NET 3.0, which is the framework thetest object is built upon.

RobustnessTestComplete 4 is very stable. Although the few times the program crashed, no information inthe project was lost.

FunctionalityA very useful feature of TestComplete is the possibility to browse all objects, methods, proper-ties and fields in basically every application based on the .NET framework. This information ispresented in very easy to use tree list view. This makes really easy to find objects and methodswithin the application.

Ease of InsertionTestComplete has a rather easy installation procedure, but nothing out of the ordinary. How-ever, the supporting tool TestExecute allows test scripts to be run on a user’s or customer’scomputer without having TestComplete installed, which is a quite useful feature.

Quality of SupportThere is an extensive documentation and solutions to various problems on Automated QA’s on-line forum. They have free mail support and access to the online message boards. TestCompleteand Automated QA is well promoted on other web sites.

CostThe enterprise version of TestComplete costs about $900 per user. The phone support costs$175 a month.

ConclusionTestComplete is stable and versatile software with the functionality currently needed for intro-ducing automatic regression testing.

Test tool 29

4.2 TestPartner 5

TestPartner [8] is developed by Compuware.

Ease of UseTestPartner is a tool easy to get started with. Many basic functions are presented clearly andare easy to locate in an intuitive user interface. With help of the documentation’s step-by-stepguide, it only takes a few minutes to set up the first test case. However, when it comes to moreadvanced functionality and larger test projects, it soon becomes very difficult and hard to stayorganized. The documentation is very limited and the samples which comes with the programonly demonstrates a small part of TestPartner’s functionality.

PowerTestPartner has a great support for data driven testing. Input data is managed through tableswith a user-friendly interface. Important features such as accessing .NET classes and objectsdoes exists in the latest version, however it is a very lengthy procedure writing test cases usingthis functionality. The scripting engine has been rewritten and in the latest version it is basedon Microsoft Visual Basic Script Engine, which is a familiar language to many testers andprogrammers.

RobustnessTestPartner did hang a few times when testing simple programs, but no data was lost.

FunctionalityAs with most test tools on the market. TestPartner lacks support of .NET 3.0 GUI testing. Noopen SDK was found to assist in the development of custom plugins. TestPartner has no nativesupport for comparing files and logs, however it is possible by writing custom scripts.

Ease of InsertionThe installation of TestPartner was simple, but nothing out of the ordinary. As the test projectsgrow it soon becomes very difficult and therefore expensive to maintain.

Quality of SupportTo access the online knowledge database one must register an account. The impressions is thatit is more difficult to find support from third party sites compared to other test tools.

Cost$9.000 per user, including 12 months of support. Supporting costs was not found on web page.

ConclusionTestPartner is a powerful test tool. Writing advanced test scripts is difficult and time demanding.

30 Test tool

4.3 AutoMate 6

AutoMate [9] is developed by Network Automation Inc.

Ease of UseThis program is really easy to use. Just after looking at some of the examples that comes withthe installation one can learn it quite fast. The method of creating test cases consists of draggingand dropping different tasks to the main script, which is really easy for a beginner.

PowerAutoMate 6 is very powerful program. It can identify every object on the screen, even if theobject is based on the .NET 3.0 framework. This makes it very useful to create test cases.

RobustnessLike other applications, AutoMate 6 is a stable software and test projects do not get lost if theprogram somehow crashes, which happens hardly ever.

FunctionalityWhat AutoMate 6 gains in power, it lacks in functionality. Although it has support for findingobjects on the screen and identifying them, it is hard to use them in test scripts due to the factthat the program has a hard time identifying them as unique. This makes it hard or almostimpossible to run more advanced test scripts. The recording function is also very inferior. Itrecords every mouse move and it is impossible to alter the recorded script without doing thewhole recording all over again.

Ease of InsertionAutoMate 6 has an easy installation procedure and comes with a service that can be modified torun test scripts upon actions in the operating systems, such as when starting a certain softwareetc.

Quality of SupportAutoMate 6 comes with great support to get started and even more advanced topics online.

CostThe professional edition tried and evaluated costs about $1000.

ConclusionIn a conclusion one can say that AutoMate 6 is well suited for simpler tasks such as startingdifferent applications, sending keystrokes, mouse clicks etc., but when it comes to more advancedtests AutoMate 6 do not have the functionality needed to implement these.

Test tool 31

4.4 QuickTest 9

QuickTest [10] is developed by Mercury.

Ease of UseQuickTest is very easy to use and has an interface easy to understand. It is effortless to recordscripts and follow the recorded scripts. It has both a task view and a script view which is veryuseful. However, it is hard to find information in the application and huge test projects are verydifficult to maintain and get an overview of. It is also difficult to change the test script sincemapping of objects on the screen is done automatically and do not always work in a satisfyingway.

PowerQuickTest has a powerful keyword driven language which means that an object deep down inthe hierarchy of an application can be mapped with a short name without the test script authorhaving to know the path to the object. The recording functionality is also easy to learn andmaintain.

RobustnessNo crashes was experienced during the evaluation period.

FunctionalityAlthough the recording was easy to do, the playback did not always work. Since the mapping ofobjects is done automatically the software sometimes has problems of identifying some objectsas unique. It also have very limited or no plugin support.

Ease of InsertionThe program is very advanced. If a common database for error logs is to be used, another softwaremust be installed as a web service. This makes it a little bit too advanced for the current needsat Beijer Electronics AB. The installation procedure crashed when trying to install the .NETFramework 1.1.

Quality of SupportIt is one of the most widely used test tool which makes it well documented on different web sites.However, it is difficult to find information about the support available from Mercury.

CostQuickTest 9 professional, which was the software tested and evaluated costs about $5.000.

ConclusionQuickTest is a popular and commonly used test automation tool. It can be hard to motivatesuch a licence cost for a small company or organization.

32 Test tool

4.5 Summary

In this chapter four different test automation tools has been reviewed. As already concluded itis important to choose a test tool fitting the company’s needs. In chapter 6 a test tool is chosenbased on the conclusions made in this chapter. In other words, this chapter only gives a reviewof each automation tool and in chapter 6 the tools are summarized and the most appropriate isrecommended.

Chapter 5Scripting Techniques

According to Fewster [2] there are several measures to be taken into consideration when in-troducing a cost and resource effective test environment. One of the more important parts isthe choice of an appropriate and well defined scripting technique. A scripting technique is thestructure and implementation of the test scripts. It is important to be consequent in the use ofa scripting technique. In this way the test scripts are ensured to be maintainable. If differenttechniques are used, not only is the code difficult to maintain, it is almost impossible for anotherperson to take over the project, and of course this is not desirable.

The higher the level of ambition the organization has for the test structure, the cost in timeand resources to introduce automatic regression testing will increase. A simple test structureis easy to implement but will not be very cost-effective as the number of test cases increasesand the project evolves and changes over time. A well defined test structure, however, may bedifficult to introduce but a lot easier to maintain over time and stays effective throughout thewhole test process.

To illustrate this, take a test case written/recorded in strictly linear scripting. This takesabout 5 times longer than doing the test manually . However, shared and data-driven test casescan take up to 40 times longer than manual testing . Looking at a long perspective the secondalternative is much more efficient since there is no need to rewrite an entire script as the projectevolves. The principle of each scripting technique is described in the sections below. [2]

33

34 Scripting Techniques

5.1 Linear scripting

Linear scripting can basically be described as when every test case is written in its own scriptfrom the beginning to the end. No scripts are reused and all code is static. Often a linear scriptis the result of recording a script performed manually. The source code below is an example oflinear scripting which creates a new project in the software Neo, that is the test object at BeijerElectronics AB. As seen, the code is difficult to read and maintain.

Dim w1Dim p1Set w1 = NameMapping.SysSet p1 = w1.NeoSet w1 = p1.MainFormCall w1.WinFormsObject("m_RibbonControl").WinFormsObject("RibbonStrip", "", 3).Click(27, 28)Call w1.WinFormsObject("MenuPanel", "").Click(38, 33)Call p1.SaveDialog.No.ClickButtonSet w1 = p1.Window("HwndWrapper[NeoIDE.exe;;d73fd5fd-23cf-4e2d-b933-eeb5a371ea05]", "New Project Wizard")Call w1.Click(449, 527)Call w1.Click(460, 530)Call w1.Click(707, 524)

Furthermore, no error checking or unexpected events are taken into consideration when usinglinear scripting.

5.2 Structured scripting

Linear scripting seldom requires programming skills, since it often is the result of a recording.Structured scripting calls for some basic knowledge in any of the most basic scripting languages.Since most test tools support syntax for the most popular scripting languages today, it is fairlyeasy to write structured test cases if the tester is familiar with any programming language.

Structured testing is basically test cases which adapts to different situations. These situ-ations is either defined or created by e.g. if-, for- and while-blocks. Used in an appropriateway, the quality of test cases can be increased considerable. Structured scripting can be usedfor example to verify that a specific message box is displayed on the screen or to repeat certaintasks multiple times. This can be useful when stress testing a program or handling unpredictedevents. An example of structured scripting is illustrated in the code below.

Dim w1Dim p1Set w1 = NameMapping.SysSet p1 = w1.NeoSet w1 = p1.MainFormCall w1.WinFormsObject("m_RibbonControl").WinFormsObject("RibbonStrip", "", 3).Click(27, 28)Call w1.WinFormsObject("MenuPanel", "").Click(38, 33)if SaveDialog.Exists then

Call p1.SaveDialog.No.ClickButtonend ifSet w1 = p1.Window("HwndWrapper[NeoIDE.exe;;d73fd5fd-23cf-4e2d-b933-eeb5a371ea05]", "New Project Wizard")Call w1.Click(449, 527)Call w1.Click(460, 530)Call w1.Click(707, 524)

Scripting Techniques 35

5.3 Shared scripting

The introduction of shared scripting allows a tester to reuse already existing code in severaldifferent test cases. This is achieved by dividing each test case in smaller tasks. Each task iscategorized and structured in a test library. These tasks can then be called from a test script tobuild up a whole test case.

The main advantage of shared scripting is that they can easily be adapted to architecturalchanges, since it is only necessary to alter one task to make changes in all scripts using that task.To illustrate this, imagine a couple of hundred test cases using the task ”Create New Project”.If the program changes so that the task does not work anymore, it is only necessary to changethe task in one place and it will affect all test cases using that task

Shared scripts may be more time-consuming and more difficult to write, but in the long runas the program changes, the maintenance and cost will be much less as if linear and structuredwould be used. An example of shared scripting can be viewed in the code below. As seen, everytest case uses predefined functions that are reused when they are needed. The advantage is thatthe method CreateNewProject only needs to be altered in one place if any changes are required.

sub TestCase1CreateNewProjectCreateButton(50,50)SaveProjectCloseProject

end sub

sub TestCase2CreateNewProjectCreateButton(100,20)CloseProject

end sub

sub CreateNewProjectDim w1Dim p1Set w1 = NameMapping.SysSet p1 = w1.NeoSet w1 = p1.MainFormCall w1.WinFormsObject("m_RibbonControl").WinFormsObject("RibbonStrip", "", 3).Click(27, 28)Call w1.WinFormsObject("MenuPanel", "").Click(38, 33)if SaveDialog.Exists then

Call p1.SaveDialog.No.ClickButtonend ifSet w1 = p1.Window("HwndWrapper[NeoIDE.exe;;d73fd5fd-23cf-4e2d-b933-eeb5a371ea05]", "New Project Wizard")Call w1.Click(449, 527)Call w1.Click(460, 530)Call w1.Click(707, 524)

end sub

5.4 Data Driven scripting

Data Driven testing allows the user to define all test input data in separate text files, e.g. sizeand layout of different components. This makes it possible to run several tests using only onetest script but with different data as input. For example, a test that put fifty components onthe screen with different positions and sizes can be run this way and later also be verified usingthe same input data. This is illustrated in Figure 5.1.

36 Scripting Techniques

open x y w h save & closeproject1.neo 20 40 200 200 project1.neoproject1.neo 30 60 180 150 project1.neoproject1.neo 40 80 160 100 project1.neoproject1.neo 50 100 140 50 project2.neo

Figure 5.1: Data File 1

The script using this data file opens a project, places a button at the coordinates x, y withthe width w and height h and then saves the project and closes it. This data file representshaving a test script running four different tests with different data.

Data Driven testing can also be taken to the next level by building in some knowledge ofthe application into the data file. This is done by representing different fields in the data file bya certain function or a type of input. An illustration of this is shown in Figure 5.2.

The script using this data file checks weather to open a new project, create a new buttonwith specific position and size, or save and close the project. In Figure 5.2 the test script opensa new project called project1.neo, then creating a button at 20, 40 with the width 200 and theheight 200, and then saves and closes the project. The next step opens the same project, butthis time three different buttons with unique positions and sizes are created and then the projectis saved under a different name.

open x y w h save & closeproject1.neo

20 40 200 200project1.neo

project1.neo30 60 180 15040 80 160 10050 100 140 50

project2.neo

Figure 5.2: Data File 2

The main difference can be described like this. Data file 1 describes how the script shouldperform different actions, but date file 2 both describes when and how the script should performthese actions.

5.5 Keyword Driven scripting

The final step towards a well structured test environment is the introduction of Keyword Driventesting. The principle is the same as in Data Driven testing with the main difference thatthe test tasks is specified with keywords rather than predefined in columns as in Data Driven

Scripting Techniques 37

testing. The main advantage of Keyword Driven testing is that a tester does not have to havegreat programming skills to write test cases. However, the supporting scripts, i.e. the scriptscorresponding to different keywords, still have to be written by an experienced programmer. Anexample of how test cases are written is illustrated in Figure 5.3.

OpenProject project1.neo CreateButton 20 40 200 200 SaveAndCloseProject project1.neo

Figure 5.3: Keyword Driven script

Each keyword (OpenProject, CreateButton, SaveAndCloseProject) in this script correspondsto a function in another script where the other words on each line are sent as parameters.

Another advantage of keyword driven testing is that since the test cases are written in apredefined language it is possible to change the whole test environment, e.g. OS, test tool,scripting language. Of course it is then necessary to change the functions that the keywordsrefer to, but the test cases can be kept intact.

5.6 Summary

In this chapter several different levels of scripting techniques has been reviewed. As already con-cluded it is important to understand different ways of implementing test cases and be consequentwhen coding. In chapter 6 an appropriate level of scripting is chosen based on the conclusionsmade in this chapter. In other words, this chapter only gives a review of each scripting techniqueand in chapter 6 the techniques are summarized and the most appropriate is recommended.

38 Scripting Techniques

Chapter 6Suggestion

It is intended to give a full proposition of the implementation of the testing strategy. Thisincludes choice of reference model, test tool, test structure and scripting technique. The propo-sition is handed out to all involved managers in the testing and development departments. Thistogether with a practical demonstration of a prototype is a foundation, that managers will givefeedback on. This feedback is considered and discussed with all involved parties. When allparties agrees on a final test strategy and environment, this is implemented and presented.

This chapter contains information on how decisions are made regarding reference model, testtool and scripting techniques. The proposition can be found at Department of CommunicationSystems at Lund University [6]. The last section in this chapter describes the feedback receivedand the final changes made to the proposition.

6.1 Reference model

As explained in section 3.1 TMM is a reference model suitable for larger organizations. Followingthe improvements recommended in the framework is very difficult, since the improvements stepsis very large. This is also sometimes very unmotivating since improvements made in the testdepartment is not traceable in the maturity scheme, see Figure 3.1.

An interesting test framework is the MTPF, section 3.3. This test practise aims towardssmaller companies and the improvement steps are smaller than TMM which makes it easier fora company of Beijer’s size to follow the test practice framework. The main disadvantages withMTPF seen from a Beijer perspective, are that improvements are executed in parallel companyexpansion. Looking at Figure 3.3, Beijer Electronics AB currently is somewhere between 2 and3. However, there is no plans to hire new employees, which makes the test framework difficultto follow. It is a disadvantage that Beijer Electronics AB already has reached phase two of thistest framework, since this leaves little room for further improvements. Finally, none of the testor developer managers are familiar with MTPF, which makes training and understanding of thereference model necessary.

The last of the evaluated improvement models is TPI, see section 3.2. This framework offersimprovements to be made in small steps, which is considered valuable when used in a smallercompany, that have limited resources and works continuously to improve the test environment.According to the results of the interview and questionnaire in Section 2.3 TPI is already familiarto the test managers and to some extent the testers, which makes it easier to apply the model.In addition to this, TPI is already used as a test improvement model at Beijer Electronics AB,

39

40 Suggestion

which is a very strong argument for this test practice framework.

Key Area Scale 0 1 2 3 4 5 6 7 8 9 10 11 12 13

Test strategy A B C D

Lifecycle model A B

Moment of involvement A B C D

Estimating and planning A B

Test-specification techniques A B

Static test techniques A B

Metrics A B C D

Test automation A B C

Test environment A B C

Office environment A

Commitment and motivation A B C

Test functions and training A B C

Scope of methodology A B C

Communication A B C

Reporting A B C D

Defect management A B C

Testware management A B C D

Test process management A B C

Evaulation A B

Low-level testing A B C

Controlled Efficient Optimizing

Figure 6.1: TPI Maturity Matrix for the organization

The TPI Maturity matrix, as seen in Figure 6.1 is a representation of how well establisheddifferent areas of the automatic test environment are. This is derived from the interview andverified by test manager Fredrik Persson. As seen in the matrix there is still a need for someimprovements, especially in those areas that does not fulfill the requirements for the lowest levelA.

The conclusion is that TPI is the most suitable test framework for Beijer Electronics ABand thereby the recommended choice of reference model for this study.

6.2 Test tool

The test software TestPartner is, as already been established, a versatile tool. However itsoon becomes to complicated when working in larger test projects. Another factor is that itis expensive both to purchase and to maintain. As seen in Figure 4.1 TestPartner along withQuickTest receives the lowest overall rating of the test tools.

QuickTest is very advanced, but far from flawless. It has inconsistent object identificationand lacks the ability to browse objects in a hierarchial tree structure, which this study found to beone of the most helpful features when creating test cases for graphical user interface applications.

AutoMate 6 is an interesting candidate. It has a really reasonable learning curve and testcases are implemented in a very intuitive way. However, it lacks support of .NET objects

Suggestion 41

which is important for this study. When doing more advanced test cases it soon becomes quitedifficult to maintain them and it is desirable to have more script support than what exists today.AutoMate 6 although receives a quite high overall rating but its functionality is not enough forthe application to be used in the test environment at Beijer Electronics AB.

TestComplete has two major advantages over the other evaluated applications. Firstly it gotthe highest overall rating of the tested program and secondly it had already been purchased byBeijer Electronics AB before the start of this study. However, this was not not a factor when thetest tools were evaluated. Normally the second advantage should not be considered, but BeijerElectronics AB has limited resources and this combined with the high purchase cost of the othertools makes this a factor to be considered. TestComplete has all the features valued high by thetesters, such as .NET support and script support. It has also the ability to browse objects in ahierarchical tree structure, which has been explained earlier to be very useful.

The conclusion is that TestComplete is a really strong candidate that will cover the needsfor a functional test environment, and therefore the recommended choice of test automation toolfor this study.

6.3 Scripting Technique

As concluded in Chapter 5, linear scripting takes the smallest amount of time to implement.Structured, shared, data-driven and keyword-driven scripting, each takes more and more timeto implement. Looking at maintenance the scale is reversed, where keyword-driven testing isthe easiest and less resource costing alternative. This means that when the test scripts and testcases has been developed, keyword-driven testing soon becomes the most valuable test structure.

Another factor to consider is that linear scripting requires practically no programming skillsand very limited testing experience. When using keyword-driven testing, experience in program-ming is a must.

Since the intent of the introduction of automatic testing is very long termed, keyword-drivenscripting is the most useful and beneficial test structure and is therefore the recommended choiceof this study.

42 Suggestion

Chapter 7Implementation

This chapter presents a prototype, which is to be considered as the base for the implementationof automatic regression testing at Beijer Electronics AB.

First section covers the recommended structure of test cases and scripts to be used in thetest environment. Next section gives a deeper explanation of the controller script, which is themain scripts that controls and runs all test cases. Finally the implemented test environment isevaluated.

7.1 Overview

This section describes an overview on how test cases are implemented and structured. As apractical example, Beijer Electronics’ project Neo has been used.

There is several considerations to be made when designing the test case structure. It is ofgreat importance that the structure, i.e. how the test cases are organized, is well defined andclear. This makes the maintenance a lot simpler and new test cases can easily be added. Theproposed Test Structure is described in Figure 7.1.

43

44 Implementation

Supporting Scripts, catagorized by function

Test projects, each corresponding to a module in Neo

Device Library GUI Library Project

LibraryDesign Library

WindowManager

ToolboxManager

UndoManager

CommandFacade

ScriptManager

ScriptEditor

SocketServer

SocketClient

ToolbarManager

LinkLibrary

Test Cases

Test Cases

Test CasesController

Keyword Driven Testing

Figure 7.1: Test Case Structure

7.1.1 Test Cases

The test cases are written in a very simple keyword-like language, preferable in a well knownenvironment like Excel, for example as illustrated in 7.2. The advantage of keeping the testcases simple and well structured is that testers with virtually no programming skills still canmaintain and write these test cases. The test cases are then interpreted by the control script inthe controller.

As seen in the Figure 7.2, the test case exists of several predefined commands, i.e. keywords,and one or more possible parameters. These should be easy to understand and well documented.

Implementation 45

Figure 7.2: Excel Test Case

The test program used in the prototype, TestComplete, offers great support for Excel withbuilt in functions for reading worksheets. Since Excel is a well known and easy to use program,it is a good reason for choosing it as a test case writing tool.

Test Cases Test Cases Test Cases

Script files Script files Script files

Figure 7.3: Test Case Structure

The test cases are categorized depending what they are designed to perform. It then becomeseasier to find and alter a specific test case and to run scripts testing certain functionalities in

46 Implementation

Neo.

7.1.2 Controller

The purpose of the control script is interpretation of test cases. The controller scans throughpredefined test cases row by row and calls the functions associated with each keyword along withparameters specified. These functions are located in the supporting scripts.

As seen in Figure 7.4, the controller consists of a control script, a configuration file and afunction reference file. The configuration file defines which test cases to execute and in whichorder. The control script is the main script that interprets the predefined test cases and usesthe function reference file to call functions in the supporting scripts corresponding to differentkeywords. There are also documentation files explaining how to use the files in the controllerlibrary. A deeper explanation of this is found in Section 7.3.

Controller

Functionreference

Config Control ScriptDocumentation

Figure 7.4: Controller Structure

7.1.3 Supporting Scripts

The supporting scripts are divided into different categories depending on what type of function-ality is being tested. For example, all functions handling the project are located in the ProjectLibrary. It is important to point out that the functions in these functional libraries do notcall internal methods in Neo. Instead it calls functions in the scripts in the test projects, asillustrated in Figure 7.1. The main reason for this is that if several different functions in thesupporting script uses the same internal method in Neo, and the test for this method needs tobe changed, i.e. due to program changes, it is necessary to change in more than one place. Themaintenance then becomes much more difficult.

Implementation 47

Library

Script filesDocumentation

Utilities

Third party utilities

Figure 7.5: Supporting Script Structure

7.1.4 Projects Scripts

Each module in Neo, relevant to the test environment is represented by a corresponding testproject. By using this structure, a change in any module in Neo that require a change in thetests, can easily be handled since the change only need to be carried out in one place. Asdescribed earlier, a structure with a more direct approach would require changes in multiplescripts.

Test Project

Documentation Script files

Figure 7.6: Test Project Structure

The test projects are the only projects that call internal methods in Neo and each projectonly call methods within the same module or same group of modules. This makes it easy toknow which project is responsible for which part of Neo and if a change is needed, it is effortlessto locate areas to change.

7.1.5 Summary

The test structure has been developed with the consideration that also testers without or withlimited programming skills are qualified to write test cases. At least one programmer is needed

48 Implementation

to write the supporting scripts and project scripts in parallel with the development of Neo.

7.2 Test Support

There are several considerations to be taken into account when testing a graphical user interfaceautomatically. Since Neo is in an early stage of development it is most likely to change inappearance and structure over time. This is illustrated in Figure 7.7 and Figure 7.8. Thescreenshots differs in both layout and functionality, and this is only over a period of six weeks.Obviously, this makes it difficult to do automatic GUI testing, e.g. by recording mouse eventsand coordinates.

Figure 7.7: Screenshot of Neo build 920

Implementation 49

Figure 7.8: Screenshot of Neo build 736

Another consideration to be taken into account is that Neo is built upon .NET Framework3.0 which in the present is very recently released and therefor it does not yet exists any fullydeveloped commercial test tools with full support for this platform. This means that the toolof our choice, TestComplete, does not recognize objects, such as components on the screen andtoolbar buttons. It makes it almost hopeless in relaying on mouse clicks and coordinates whenit comes to automation testing of the GUI.

It is therefore vital to be able to get under the shell, i.e. by directly accessing internal methods

50 Implementation

and properties. In this way, the testing process is not sensitive to graphical user interface designchanges, but only to major architecture structure changes, which are not very common. Thisis mainly due to facts that the developers uses an architecture design called SOA Light, ServiceOriented Architecture Light. SOA is a design approach where the project is based on numerousdifferent modules, which all has its own responsibility. These modules communicate with eachother through services, which is a certain interface and is independent of how the other modulesare implemented.

To be able to access the functionality of each and every service, there has to be some sort ofcommunication between the test program and the test object. Due to the fact that TestCompletehas great support for .NET applications and Neo is built upon the .NET framework the processof accessing methods and functionality from Neo is significantly simplified. TestComplete allowsthe user to access properties and internal methods of Neo directly out of the box. However, byadding a property containing a handle to a helper class the access to properties is simplified evenmore.

The helper class is a class which only responsibility is to provide simple methods to performmore advanced tasks, e.g. accessing objects on the screen, handling projects and calling internalmethods helpful when writing test cases. Due to lack of time, the developers have not hadthe possibility to fully implement the helper class. This has limited the number of test casespresented. It is very important to allocate sufficient time and resources for the development oftest support in the software. Also important to say is that a continuous development of the testsupport is needed to ensure that the automatic regression testing is at a functional and efficientstate at the same time as the software.

7.3 Controller Script

The controller script is the most essential part of the implementation of the test structure. Thisscript loads and executes all test cases, defined in a single Excel file, see Figure 7.9. A test caseis specified in an Excel file which is interpreted by the controller script and a correspondingmethod in the supporting scrips is executed.

The Test Config file contains all test cases which is to be run. First column simply definesif a specific test case is to be executed or not. If this column contains a #1 the test case isskipped. Next column specifies where the test case file is located, i.e. path to folder. The reasonfor having the possibility to place various test cases in different folder is that the test cases staysstructured as the number of test cases increases. Third and last column, contains the actual testcase to be run. This name corresponds to an Excel file in the specified folder.

1The universal standard sign for defining comments

Implementation 51

Figure 7.9: Config file

As seen in Figure 7.10, the Test Cases are very intuitive and simple to read and write. Thefirst column always contains the keyword to be executed. The following columns specifies theparameters to be sent with the keyword. For example, looking at the figure, we see that the firstcommand is CreateButton, which intuitively creates a button. The button is named ”Button1”and also gets the caption ”Button1”. And is created at position 50 (x), 50 (y). All of theseparameters is specified in an reference file and a more extensive documentation.

Figure 7.10: Test Case file

All existing keywords and the corresponding namespace2 is defined in a the FunctionalReference file, see Figure 7.11. It is a useful resource for test case writers, which needs a list ofavailable keywords.

2Defines the location of a function

52 Implementation

Figure 7.11: Functional Reference file

7.4 Summary

As seen in this chapter a foundation for the test structure is presented. The main idea is tokeep the test cases as simple as possible which allows testers with virtually no programmingexperience to write test cases. Only one experienced programmer is needed to maintain andevolve the test structure with new functionality in parallel with the development of Neo.

Chapter 8Conclusion

This study has concluded that TPI is a good choice for smaller organizations that continuouslywork for a better test environment. It is strongly recommended to review the TPI Matrix afterthe introduction of automated regression testing at Beijer Electronics AB. As automated testingbecomes a part of the already existing test environment, it is important that automated testingreaches the existing maturity levels, or else the TPI matrix must be reconsidered. TPI encour-ages a well defined and structured test environment. Since Beijer Electronics AB already hasestablished parts of the TPI process, the introduction of automated testing should be relativelyeasy to carry out.

Among the test tools evaluated, TestComplete is by far the most versatile and useful tool forBeijer Electronics AB. It has also already been purchased and is somewhat familiar to the testersat the company. The development department is already using software from AutomatedQA forbuilding their projects. This makes the task of running selected automated tests with everybuild of Neo very easy. This saves time for the development department since defects of the mostcommon functionalities can be detected immediately. TestComplete also offers the functionalityneeded for the current levels of maturity in the TPI matrix, such as exporting test logs and ofcourse the automation of tedious tasks.

The test environment prototype designed for the company is based on keyword driven testing,which is considered the most cost efficient way of implementing test cases. However, this requiresa tester with some programming skills for maintaining and evolving test scripts. The advantageis that testers writing test cases do not need any programming skills.

The prototype is designed in a similar structure to Neo, which means a module based design,where different test scripts corresponds to certain modules in Neo. This makes it easy to maintainand update the test scripts as Neo evolves. It has been verified that this structure works wellwith TestComplete. The amount of time needed for writing test scripts was even lower thanexpected. It is easy to find and alter different methods in test structure to comply with thechanges made in Neo between builds.

It is important to realize that not all testing can be done automatically. There is still needfor some manual testing, such as verifying that the design is user friendly. Certain graphicalfunctionalities are also very hard to test automatically. For example it is difficult to prove thata zoom function or changing font type actually works.

53

54 Conclusion

8.1 Proposition evaluation

The feedback received from the proposition has been overall positive but somewhat limited. Thefeedback from the questionnaire and interview was very good, with many valuable comments.However, the feedback from the proposition was much lower, only one person gave constructivecriticism. It is apparent that the importance of receiving constructive feedback should have beenmuch more emphasized. The lack of feedback makes it difficult to get an overview how the testenvironment actually is going to look in its final state. Any future improvements or changesto the test environment is made by employees at Beijer Electronics AB. It is understandablethough, that is it difficult to give constructive criticism when automated regression testing is anentirely new concept at the company.

8.2 Prototype evaluation

The prototype presented has limited functionality. The reason for this is partly because of thefact that prototypes seldom has extensive functionality, but also due to the outline of this study.The idea was to realize the test environment in parallel with the development of Neo. This masterthesis decreased in priority due to narrow deadlines in the development cycle. This lead to tothat the support needed to implement a larger test structure and more test cases could not beallocated. A natural measure for us was to focus on the theory and give the documentation andtest environment needed for Beijer Electronics AB to evolve and complete the test environmenton their own.

During the last weeks of work a beta version of TestComplete 5 was used, mainly because ofthe support of Windows Presentation Foundation objects (.NET 3.0 Framework) [11]. Since Neois built upon .NET 3.0 Framework this feature is invaluable. The downside is that the currentversion is far from stable, but looking at previous versions of TestComplete a stable version canbe expected in the near future.

We have not experienced any limitations with the test environment prototype. It is easyto add new functionality and write new test cases. However, already in this early stage ofdevelopment the importance of updating and maintaining test scripts has become apparent. Ifnot keeping track of changes in Neo it soon becomes very difficult to maintain a high quality oftest scripts.

8.3 Future improvements

Although it is not a part of this master thesis, a version control system, e.g. CVS and ClearCase,is a valuable addition to the test environment. One reason is for example if the developmentdepartment decides to go back to an earlier build of Neo. It is clear that the test scripts writtenfor that specific build should be used.

Another useful feature is the possibility to run a number of specific tests automatically withevery build. Currently Automated Build Studio [?] is used at Beijer Electronics AB. This softwarehas great support of running TestComplete projects and scripts.

The importance of error checking and logging can not be disregarded. These are the mainfactors of increasing stability of the test environment and simplifying debugging and maintenanceof test scripts. This is especially important as the test environment grows but it is also important

Conclusion 55

to know that increased functionality of error and logging handling can always be improved.

56 Conclusion

References

[1] Burnstein I, Practical Software Testing, Springer: Chicago, IL, 2003

[2] Fewster M, Graham D, Software Test Automation: Effective use of test execution tools,Addison-Wesley, 1999

[3] Pol M, Teunissen R, Veenendaal E van, Software Testing: A guide to the TMap Approach,Addison-Wesley, 2002

[4] Pol M, Koomen T, Test Process Improvement: A practical step-by-step guide to structuredtesting, Addison-Wesley, 1999

[5] Karlstrom D, Runeson P, Norden S, A Minimal Test Practice Framework for EmergingSoftware Organisations, Software Testind, Verification and Reliability, 2005

[6] Lund O, Stromqvist C, Proposition of Automatic Regression Testing for Beijer ElectronicsAB, 2006http://www.telecom.lth.se/

[7] TestComplete, January 22, 2007http://www.automatedqa.com/products/testcomplete

[8] TestPartner, January 22, 2007http://www.compuware.com/products/qacenter/375 ENG HTML.htm

[9] AutoMate, January 22, 2007http://www.networkautomation.com/automate/

[10] QuickTest, January 22, 2007http://www.mercury.com/us/products/quality-center/functional-testing/quicktest-professional/

[11] Windows Presentation Foundation, January 22, 2007http://wpf.netfx3.com/

57

58 References

Appendix AQuestionnaire

A.1 Organization Stage

To help us introduce a testing strategy well compatible with Beijer Electronics, we need you toanswer a few questions regarding the organization.

Which best describes your current position?Check the best alternative· Software Quality Assurance Group Leader· Software Quality Assurance Group Member· Software Engineer (developer)

What is your current test-related duties?Check one or more alternatives which applies to you.· Test policy and goal development· Test planning· Test case and test procedure design· Test execution· Maintenance of case database· Standards development· Reviews and audits· Status tracking· Test training· Hiring and recruiting· User/Client communications· Test process control· Test process evaluation· Test process improvement· Defect prevention· Usability testing· Tool evaluation

59

60 Questionnaire

Are you familiar with any of the following test practise frameworks?Check the test practise frameworks you are familiar with.· TMM (Test Maturity Model)· TPI (Test Process Improvement model)· TMap (Test Management approach)

What is the extent of your experience in software testing?· Less then 1 year· 1 to 2 years· 2 to 3 years· More than 3 years

How would you generally characterize the nature of your testing process?Check the alternative that is the best description· Ad Hoc· Informal· Somewhat structured· Highly structured

Questionnaire 61

A.2 Tool Evaluation

Rate the importance of properties of the tool by grading them from 1 (not important) to 7 (veryimportant).

· Ease of use· Power· Robustness· Functionality· Ease of insertion· Quality of Support· Cost

Ease of use - The tool should be easy to learn, have a predictable behavior, deliver meaningfulfeedback to the user, have adequate error handling capabilities.

Power - The tool should have numerous of useful features, have the possibility to interpret the.NET standard, perform validation checks on objects or structures.

Robustness - The tool should be reliable, recover from failures without major loss of informa-tion, evolve and retain compatibility between old and new versions.

Functionality - The tool should feature several layers of the test strategy, e.g. test planning,test execution, test management.

Ease if insertion - The tool should be easy to incorporate into the organizational environment,not require any previous experience or training to be able to use, have results available to theuser without a long set-up process, run on the operating system used by the organization.

Quality of Support - The tool should offer a solid manual. The vendor should offer mainte-nance, installation, training, phone and mail support.

Cost - The tool should have a reasonable cost for maintenance, purchase and installation.

62 Questionnaire

A.3 Tool Features Evaluation

Rate the importance of features in the tool by grading them from 1 (not needed) to 5 (essential).

· Remote handling· Recording· Native .NET testing· GUI Testing· Script support· Plug-in support· API support· COM support· Distribution of test reports· Compare test results· Other

Remote handling - The test tool should be able to execute tests and review reports from an-other computer.

Recording - The test tool should offer the possibility to record mouse and keyboard handling.

Native .Net testing - The test tool should offer native .NET support, i.e. having full accessto internal methods and objects within all applications written with the .NET framework as abase.

GUI Testing - The test tool should offer the possibility to test a Graphical User Interface.

Script support - The test tool should offer the possibility for the user to write advanced testcases using any arbitrary scripting language.

Plug-in support - The test tool should offer the user the possibility to write own and downloadplug-ins to extend the functionalities of the test tool.

API support - The test tool should support sending and receiving Windows API calls.

COM support - The test tool should support sending and receiving COM objects.

Distribution of test reports - The test tool should be able to automatically distribute testreports by email or submit results to database.

Compare test results - The test tool should be able to compare different test runs and identifyany differences.

Other - Write down features, not already in the list, you think are important.

Questionnaire 63

A.4 Identifying Test Cases

The main focus on this Test Project is regression testing. Except for identifying regression bugs(functionality testing), what du you consider are important test areas to include?

Rate the importance of test areas by grading them from 1 (not needed) to 5 (essential)

· Execution time of Neo· Execution time of generated binaries· Response time of Neo· Response time of generated binaries· Memory usage of Neo· Size and memory usage of generated binaries· Memory leaks in Neo· Memory leaks in generated binaries

Execution time - Evaluating the execution time (startup time) of the application.

Response time - Evaluating the response time of different functionalities in the application.

Memory usage - Evaluating the memory usage of the application.

Size and memory usage - Evaluating the size on the disk and the memory usage of the ap-plication.

Memory leaks - Test if there is any memory leaks in the application.

64 Questionnaire

A.5 Programming language

Different test tools supports different scripting languages to create test cases. Therefore it isessential to know which (if any) scripting language you prefer.

Check the scripting languages you prefer.

· VBScript· JScript· C#Script· DelphiScript

To help us choose a test tool reflecting your needs, it’s vital to know the testers programmingskills, since certain test tools uses some kind of scripting language to create test cases.

Rate your programming skills by grading them from 1 (non-existing) to 5 (expert).

· Visual Basic· Java· C#· Delphi

Questionnaire 65

A.6 Test/Trouble Reporting

Our goal is to present well structured and detailed reports. Except from this, what do youconsider are important features of the test/trouble reports (e.g. exporting to different formats,automatic submission online, etc.)?

66 Questionnaire

A.7 Final Comments

Our goal is to provide a complete testing system that suits the employees and organization ofBeijer Electronics. If you have anything to add to this questionnaire. Please do so in this section.

Appendix BInterview Questions

This chapter contains the questions asked during the interview with test manager Fredrik Perssonat Beijer Electronics AB. The question was gathered from Burnstein [1].

B.1 Organization Background

How many people are employed in the organization being assessed?· Total employees· Number engaged in software development· Number engaged in software testing

Please describe the percent of staff engaged in testing as follows?· Full time· Part time in software development· Consultants

Are the responsibilities for test process improvement clearly defined and supported?1 2 3 4 5

How is the testing group organized? (please select one)· Developers do the testing· Test group within development, report to project manager· Separate test group, report to test manager· Part of Software Quality Assurance group

67

68 Interview Questions

B.2 Maturity identification

According to Burnstein, the TMM process is on maturity level 2 if more than half of the levelsubgoals are satisfied. A subgoal is satisfied if more than 50% of the questions for the subgoalare answered by YES.

Subgoal 2.1Develop testing and debugging goals and policies

· Has a committee on testing and debugging been established?

· Have policies, goals, activities, and tools for the testing processbeen identified, documented and approved?

· Do the software testers follow a written organizational policy fortesting when test planning?

· Are basic measurements used to determine achievement of test-ing goals?

· Have the testing policies and goals been developed with inputsuser/client groups with respect to their needs?

· Do developers/testers log defects into the repository on a con-sistent basis?

· Are testing/debugging policies and goals periodically reviewed?

Interview Questions 69

Subgoal 2.2Initiate a test planning process

· Has an organizationwide test planning commitee or group beenestablished??

· Are there procedures for test planning?

· Is there adequate support and funding for test planning for allprojects?

· Are test goals/objectives used as a basis for test planning?

· Have test plan templates been developed?

· Are there appropriate planning tools available for test planning?

· Are testers trained properly to develop test specifications, testdesigns and test cases?

· Are test-related risks considered when developing test plans?

· Are estimates (time, budget, tools) available from past projectsfor use in test planning?

· Is test planning done at the acceptance level?

· Do testers have the opportunity to give inputs to the test planat all levels?

· Is the test planning process reviewed on a periodic and/or event-driven basis?

· Are other test-related items such as test transmittal reports, testlogs, test incident reports, and test summary reports defined inorganizational documents?

· Are other test-related items such as test transmittal reports, testlogs, test incident reports, and test summary reports completedfor each project?

· Do testers collect and store basic test-related measurements?

· Are the basic test measurements used to ensure that basic testinggoals have been met?

70 Interview Questions

Subgoal 2.3Institutionalize basic testing teqniques and methods

· Has a committee or group been formed to evaluate and recom-mend a set of basic testing techniques, methods, and tools?

· Have the recommendations of the group been documented, dis-tributed, and approved?

· Have basic tools and techniques been included in test policies?

· Have appropriate forms and templates been designed to supportbasic testing techniques?

· Have test plan templates been developed?

· Are adequate resources provided by upper management to sup-port the use of basic testing techniques and methods, as well asbasic testing tools?

· Have testers been trained to apply the basic tools, forms, andmethods?

· Are basic testing techniques, and methods applied on an orga-nizationwide basis?

· Are estimates (time, budget, tools) available from past projectsfor use in test planning?

· Are basic testing techiques and methods reviewed periodically?

· Is testing planned and implemented at multiple levels (unit, in-tegration, system, etc.)?

Appendix CImplementation

C.1 Controller script

This section contains the source code of the controller script. The source code is not final andshould be continuously developed and improved as the supporting scripts and test cases also aredeveloped. It is fully functional, but error checking and functionality can always be extended.AutomatedQA, developers of TestComplete has announced plans of a native support for key-word driven testing in a future version of TestComplete. This would possibly make the controllerscript superfluous.

This is a very rough description of how the controller script works.

1. The function reference file is loaded. This contains a list of all accepted functions whichcan be used in test cases.

2. The config file is loaded. This contains a list of all test cases to be run.

3. The controller script loads the first test case to be found in the config file

4. The keywords with parameters is found in the test case file. The corresponding functionsis executed. These functions is found in the functional libraries.

5. The controller script loads next test case file which is found in the config file.

6. Step 4 is repeated until all test cases has been run.

7. The controller script is done and the result is presented in TestComplete.

71

72 Implementation

sub main

’Initial size of functionlibrarysize = 100

’Maximum number of parameters. Max 10paramsMax = 10

’Location of the Neo test directoryaPath = "..\neoTest"

’Location of the configuration file in the Neo test directorycFileName = "config.xls"

’Location of the functional reference file in the Neo test directoryfFileName = "functions.xls"

’Loads the function reference fileset functionsFile = DDT.ExcelDriver(aPath + "\" + fFileName, "Sheet1")

’Creates an array and stores all functional referencesfReference = BuiltIn.CreateVariantArray2(0,1,0,size-1)count = 0while not functionsFile.EOF

count = count + 1if count > size then

size = size * 2BuiltIn.VarArrayRedim fReference,size-1

end iffReference(0,count-1) = functionsFile.Value(0)if functionsFile.Value(1) <> "" then fLib = functionsFile.Value(1)fReference(1,count-1) = fLibfunctionsFile.Next

wend

’Loads the configuration fileset configFile = DDT.ExcelDriver(aPath + "\" + cFileName, "Sheet1")

’Loops through the configuration filewhile not configFile.EOF

’Stores the project folderif configFile.Value(1) <> "" then fFolder = configFile.Value(1)

’Checks weather the test case should be run or notif AnsiCompareStr(VarToStr(configFile.Value(0)),"#") <> 0 then

’Stores the test case namefName = configFile.Value(2)

’Loads the test case fileset testcaseFile = DDT.ExcelDriver(aPath + "\" + fFolder + "\" + fName + ".xls", "Sheet1")

’Create new LogFolderlogFolder = log.CreateFolder(fName)log.PushLogFolder(logFolder)

’Number of parametersfParamsCols = testcaseFile.ColumnCount-1

’Creates the parameters arrayfParams = BuiltIn.CreateVariantArray(0,9)

’Loops through the test case filewhile not testcasefile.EOF

’Checks if there is a command in the first columnif VarToStr(testcaseFile.Value(0)) <> "" then

’Locate the command in the functional referencecount = 0while (not fReference(1,count) = testcaseFile.Value(0)) and count < size-1

count = count + 1wendif not fReference(1,count) = testcaseFile.Value(0) then

count = count + 1end if

Implementation 73

’Checks weather the command exists or notif (count = size) then

’Error if the command doesn’t existLog.Error("Command " + testcaseFile.Value(0) + " does not exist in the functional library")

else

’Loops through all parametersfParamCount = 0while (not testcaseFile.Value(fParamCount+1) = "") and

fParamCount < fParamsCols-1 and fParamCount < paramsMax-1fParams(fParamCount) = testcaseFile.Value(fParamCount+1)fParamCount = fParamCount + 1

wendif not testcaseFile.Value(fParamCount+1) = "" then

fParams(fParamCount) = testcaseFile.Value(fParamCount+1)fParamCount = fParamCount + 1

end if

Err.Clearon error resume next

’Calls the associated command with parameterslogFolder = log.CreateFolder(fReference(1,count))log.PushLogFolder(logFolder)’log.message "Now running " & fReference(1,count)if fParamCount = 0 then Runner.CallMethod fReference(0,count)+"."+fReference(1,count)if fParamCount = 1 then Runner.CallMethod fReference(0,count)+"."+fReference(1,count),

fParams(0)if fParamCount = 2 then Runner.CallMethod fReference(0,count)+"."+fReference(1,count),

fParams(0),fParams(1)if fParamCount = 3 then Runner.CallMethod fReference(0,count)+"."+fReference(1,count),

fParams(0),fParams(1),fParams(2)if fParamCount = 4 then Runner.CallMethod fReference(0,count)+"."+fReference(1,count),

fParams(0),fParams(1),fParams(2),fParams(3)if fParamCount = 5 then Runner.CallMethod fReference(0,count)+"."+fReference(1,count),

fParams(0),fParams(1),fParams(2),fParams(3),fParams(4)

if fParamCount = 6 then Runner.CallMethod fReference(0,count)+"."+fReference(1,count),fParams(0),fParams(1),fParams(2),fParams(3),fParams(4),fParams(5)

if fParamCount = 7 then Runner.CallMethod fReference(0,count)+"."+fReference(1,count),fParams(0),fParams(1),fParams(2),fParams(3),fParams(4),fParams(5),fParams(6)

if fParamCount = 8 then Runner.CallMethod fReference(0,count)+"."+fReference(1,count),fParams(0),fParams(1),fParams(2),fParams(3),fParams(4),fParams(5),fParams(6),fParams(7)

if fParamCount = 9 then Runner.CallMethod fReference(0,count)+"."+fReference(1,count),fParams(0),fParams(1),fParams(2),fParams(3),fParams(4),fParams(5),fParams(6),fParams(7),fParams(8)

if fParamCount = 10 then Runner.CallMethod fReference(0,count)+"."+fReference(1,count),fParams(0),fParams(1),fParams(2),fParams(3),fParams(4),fParams(5),fParams(6),fParams(7),fParams(8),fParams(9)

log.PopLogFolder

end if

end if

testcaseFile.Next

wend

end if

log.PopLogFolderconfigFile.Next

wend

end sub

74 Implementation

C.2 Supporting scripts

Each supporting script contains a number of functions to be used when writing test cases.These functions corresponds to different keywords. The scripts is categorized by its functionalresponsibilities, for example one script for all keywords relevant for project handling, anotherscript for object creation etc.

In this section a part of the functional library Project is presented. As we can see in thescript no internal methods of the software to be tested is called. All internal methods is calledfrom the project scripts. This is to ensure it is easy to locate and perform any changes needed.This means that the methods in the supporting scripts only calls methods in the project scripts.

’ Create a new project in Neosub NewProjectWizard

if not p_Miscellaneous.NeoExists thenexit sub

end ifif p_WelcomeScreen.CheckExists then

if p_WelcomeScreen.CheckVisible then p_WelcomeScreen.Hideend ifp_CommonLibrary.NewProjectif p_ProjectManager.SaveDialogExists(false) then

p_ProjectManager.SaveDialogClickNoend if

end sub

’ Open a project with path psub OpenProject(p)

if not p_Miscellaneous.NeoExists thenexit sub

end ifif p_WelcomeScreen.CheckExists and p_WelcomeScreen.CheckVisible then

p_WelcomeScreen.Hideend ifp_CommonLibrary.OpenProjectif not p_ProjectManager.OpenDialogExists then

exit subend ifp_ProjectManager.OpenDialogOpen(p)if p_ProjectManager.ReloadDialogExists then

p_ProjectManager.ReloadDialogClickNoend ifif p_ProjectManager.SaveDialogExists(false) then

p_ProjectManager.SaveDialogClickNoend if

end sub

’ Close project currently opened in Neosub CloseProject

if not p_Miscellaneous.NeoExists thenexit sub

end ifif p_WelcomeScreen.CheckExists and p_WelcomeScreen.CheckVisible then

p_WelcomeScreen.Hideend ifp_CommonLibrary.CloseProjectif p_ProjectManager.SaveDialogExists(false) then

p_ProjectManager.SaveDialogClickNoend if

end sub

’ Save current project in Neo with path psub SaveProjectAs (relDir)

if not p_Miscellaneous.NeoExists thenexit sub

end ifif not p_CommonLibrary.IsProjectOpen then

exit sub

Implementation 75

end ifp_CommonLibrary.SaveProjectAsp_ProjectManager.SaveAsDialogSave(relDir)if p_ProjectManager.ReplaceDialogExists then

p_ProjectManager.ReplaceDialogClickYesend if

end sub

C.3 Project Scripts

The project scripts contains functions used when accessing internal methods and graphical userinterface controls of the software, in this specific case, Neo. Each project scripts corresponds toa module of the software program. This makes it easy to maintain and update the scripts, dueto changes in the program. This is especially effective when the software is built upon serviceoriented architecture.

’ Constants used when locating GUI objects

SAVE_DIALOG = "Window(""#32770"", ""Save project"", 1)"SAVE_DIALOG_BUTTON_NO = "Window(""Button"", ""&No"", 2)"OPEN_DIALOG = "Window(""#32770"", ""Open Neo Project"", 1)"OPEN_DIALOG_BUTTON_OPEN = "Window(""Button"", ""&Open"", 2)"SAVE_AS_DIALOG = "Window(""#32770"", ""Save Neo Project"", 1)"SAVE_AS_DIALOG_BUTTON_SAVE = "Window(""Button"", ""&Save"", 2)"COMBOBOX_WRAPPER = "Window(""ComboBoxEx32"", """", 1)"COMBOBOX_MAIN = "Window(""ComboBox"", """", 1)"COMBOBOX_EDIT = "Window(""Edit"", """", 1)"RELOAD_DIALOG = "Window(""#32770"", ""Reload project"", 1)"RELOAD_DIALOG_BUTTON_NO = "Window(""Button"", ""&No"", 2)"

’ Checks if the Save Dialog is openfunction SaveDialogExists(exist)

if NameMapping.sys.neo.WaitChild(SAVE_DIALOG, 0).Exists thenSaveDialogExists = truemess = "Save Dialog exists"

elseSaveDialogExists = falsemess = "Save Dialog do not exists"

end ifif not (exist = SaveDialogExists) then

log.error messend if

end function

’ Checks if the Open Dialog is openfunction OpenDialogExists

if NameMapping.sys.neo.WaitChild(OPEN_DIALOG, 0).Exists thenOpenDialogExists = true

elseOpenDialogExists = falselog.error "Open Dialog do not exists"

end ifend function

’ Checks if the Reload Dialog is openfunction ReloadDialogExists

if NameMapping.sys.neo.WaitChild(RELOAD_DIALOG, 0).Exists thenReloadDialogExists = true

76 Implementation

elseReloadDialogExists = false

end ifend function

’ Enters text into a file dialog and clicks the ’Open’ buttonsub OpenDialogOpen(path)

cwd = dotNET.System_IO.Directory.GetCurrentDirectoryfile = cwd + "\..\" + pathNameMapping.sys.neo.WaitChild(OPEN_DIALOG,0).WaitChild(COMBOBOX_WRAPPER,0).WaitChild(COMBOBOX_MAIN,0).

WaitChild(COMBOBOX_EDIT,0).wText = fileNameMapping.sys.neo.WaitChild(OPEN_DIALOG,0).WaitChild(OPEN_DIALOG_BUTTON_OPEN,0).ClickButton

end sub

’ Enters text into a file dialog and clicks the ’Save’ buttonsub SaveAsDialogSave(path)

cwd = dotNET.System_IO.Directory.GetCurrentDirectoryfile = cwd + "\..\" + pathNameMapping.sys.neo.WaitChild(SAVE_AS_DIALOG,0).WaitChild(COMBOBOX_WRAPPER,0).WaitChild(COMBOBOX_MAIN,0).

WaitChild(COMBOBOX_EDIT,0).wText = fileNameMapping.sys.neo.WaitChild(SAVE_AS_DIALOG,0).WaitChild(SAVE_AS_DIALOG_BUTTON_SAVE,0).Click

end sub

’ Clicks No in the Save Dialogsub SaveDialogClickNo

NameMapping.sys.neo.WaitChild(SAVE_DIALOG,0).WaitChild(SAVE_DIALOG_BUTTON_NO,0).Clickend sub

’ Clicks No in the Reload Dialogsub ReloadDialogClickNo

NameMapping.sys.neo.WaitChild(RELOAD_DIALOG,0).WaitChild(RELOAD_DIALOG_BUTTON_NO,0).Clickend sub