automated testing using scriptingthe data can then be taken from each area to make improvements....

CEN 4072

Software Testing

Automated Testing using Scripting

Abstract

When it comes to software testing, there are many approaches to solving the problem.

One of the most widely used approaches is scripting. Using a scripting programming language,

such as Python, developers can create testing scenarios to help ensure software works as

intended. Also, these tests can be run without the need for a compiler, which makes scripting

very flexible. Exploring how to use scripting with various methods of automated testing, along

with the advantages and drawbacks of using such testing, can shed light on the overall scope of

the practice.

When scripting is the method of choice when implementing testing scenarios, utilizing

methods such as black box or white box testing can help test the functionality of many different

aspects of the program. Frameworks associated with testing and scripting languages also provide

a large basis for ensuring successful functionality. Although frameworks may provide vast

functionality, it may also be appropriate to incorporate various tools such as CIT, or continuous

integration tools, to help developers track what code has been modified within the testing

environment. Producing robust software requires the use of many different scripts, frameworks,

and tools to ensure the software works as intended.

Keywords: Automated, Scripting, Testing, Framework, Black Box Testing, White Box Testing,

Unit Testing, Integration Testing, Linear Scripting, Modular Testing, Data-Driven Testing,

Keyword-Driven Testing, Hybrid-Driven Testing

CEN 4072 – Software Testing: Essay

2

Introduction

Scripting has become a necessity in many aspects of software development. From

website building to creating some of the most useful applications, such as Dropbox and Spotify.

Languages such as Python and JavaScript have become highly desirable skills for this reason.

The impact of having a scripting or interpreted language versus a compiled language has allowed

for the expansion of technological capability. Not only can stand alone programs be built with

scripting, but scripting can also be used alongside traditional programming languages such as a C

or C++. Scripting alongside these compiled languages can have a variety of uses, but one of the

most important is testing.

Testing is often manual or automated. The manual approach tends to be tedious and

repetitive in nature which can quickly become a time-consuming ordeal. Manually testing each

individual aspect of a program is unrealistic when systems are large as there is much more room

for error. Automated testing is “the practice of running tests automatically, managing test data,

and utilizing results to improve software quality.” [8] The automated approach is where scripting

shines. It allows developers to write the step by step instructions on how to test a program into

one centralized file. This file can then be modified and used to test the system any number of

times without having to rewrite each instruction.

Using scripts, we can test a variety of areas within software that include:

• Integration - End-to-end test that checks interaction between dependencies.

• Performance - Test to determine runtime under heavy load.

• Unit Tests – Testing a single unit in isolation.

• Code Analysis - Looks for flaws in code writing.

• Debugging - Can be used to log program in search of faults.

The data can then be taken from each area to make improvements. Automation testing

drastically reduces the issues associated with manual testing. The entire goal of automation is to

save time, effort, and money. Being able to have a piece of software test your program allows for

resources to be allocated in other important areas.


3

Features of the Script Language

Syntax and Semantics

Scripting languages generally follow a simpler syntax as the languages are interpreted rather

than compiled. For instance, scripting languages like Python do not have to declare a data type

for a variable during initialization so that code like “temp = 10” would initialize variable “temp”

to an integer value of 10 automatically. This code would run without undergoing any interpreter

or run time error in Python, whereas a compile time and runtime error would be generated in a

non-scripting language such as Java. This error would occur since the data type must be

predefined for a variable during initialization, such as “int temp = 10”. Also, Java follows a

syntax in which it requires the developer to insert code in a defined main or alternative function,

whereas the main function does not have to be defined in a scripting language (e.g. “int main()

declaration in Java”).

Figure 1. Above is an example of Java which is not a scripting language as it requires a more complex syntax with semicolons,

parentheses, and predefined data types.

Interactivity

Interactive programming is defined as a procedure of writing parts of program while it is

already being run, and since scripts are interpreted instead of being compiled, it is easier for web

developers to add tools for the user interaction to the website. Examples of this would include

visual descriptions, which include background and foreground colors. Interactivity is made easier

in scripting as it does not involve external libraries which might have to be precompiled and

predefined as in the case of non-scripting languages.

Flexibility

Most scripting language can work with one another to enhance the functionality of the

program. For example, using HTML with aspects of PHP to connect it to an online database and

Django, a python wrapper, which could be used to launch the website through a server. This

feature of scripting language makes it more powerful than an ordinary programming language

each of which requires a separate compiler and distinct environments. Flexibility aspect of

Scripting language makes it idea for the developers to design websites.


4

Memory Management

Programming and scripting languages have different ways of dealing with the

dynamically and statically declared variable. For example, memory management in Python

involves a private heap containing all Python objects and data structures. The management of

this private heap is controlled internally by the Python memory manager, which deals with

various dynamic storage management aspects, like sharing, segmentation, pre- allocation, or

caching.

At the lowest level, a raw

memory allocator ensures that there is

enough room in the private heap for

storing all Python-related data by

interacting with the memory manager

of the operating system. On top of the

raw memory allocator, several object-

specific allocators operate on the same

heap and implement distinct memory

management policies adapted to the

peculiarities of every object type. For

example, integer objects are managed

differently within the heap than strings,

tuples, or dictionaries because integers

have different storage requirements and

speed and space tradeoffs. The Python

memory manager thus delegates some

of the work to the object-specific

allocators but ensures that the latter operate within the bounds of the private heap.

It is important to understand that the management of the Python heap is performed by the

interpreter itself and that the user has no control over it, even if they regularly manipulate object

pointers to memory blocks inside that heap. The allocation of heap space for Python objects and

other internal buffers is performed on demand by the Python memory manager through the

Python/C API functions.


5

Types of Automated Testing

Automated testing is a very broad term, and, as expected, there are quite a few different

types of automated testing that a developer or QA tester can use. Developers have a large

toolbox to utilize when debugging software. As with most things in life, different situations

require different tools; the same goes for automated testing; not every situation is the same, so

you should which type of testing is required for a situation.

There are two categories that most types of automated testing fall under, black box

testing and white box testing; many types of testing such as integration testing and regression

testing can be counted both as white box or black box testing depending upon the situation and,

or the needs of the tester. White Box testing is mainly focused on testing the inner workings or

functions of a piece of software, though not always. With white box testing, the tester or

developer has direct access or knowledge of the internal structure or code of the software they

are testing; due to this, white box testing is normally performed by experienced software

developers. Being able to access the code of the software to test it makes white box testing ideal

for testing individual functions, function logic, statements, and algorithms. This comes at the

downside of generally taking more time. However, it is easier to automate white box testing than

black box testing. Black Box testing allows the tester to have no access or knowledge of the

internals of the software, where the external behavior of the program is being tested. The process

can be described as testing through trial and error.

Figure 2. [17] Diagram above visualizes differences between white box and black box testing

It can be easier to view black box and white box testing as two overarching categories

that contain many different types of testing. Both white box and black box testing can be

automated, as well as the various methods or types of testing within these two categories. Some

of the more commonly automated test types are functional testing, regression testing, and

integration testing [15]. Functional testing and regression testing are typically used as a type of

black box test, while integration testing can be used for either. Functional testing is used to

mainly to verify a particular feature, normally by comparing its results against the feature’s

specifications. It’s only interested in whether the features produce the expected output.

Functional testing could include testing security, application programming interfaces (APIs), and

databases [16]. Regression testing is concerned with re-testing portions or even all parts of an

existing piece of software after a change has been made. This is to verify whether the software


6

still functions as expected, that the change hasn’t impacted the software’s functionality. As a

type of black box testing, these two tests are not made to utilize the actual code base and only

test the output. Integration tests can be considered a fuller test; instead of testing a single feature

or function, it tests a combination of features or systems to ensure they are operating correctly.

This can be exceptionally useful as it will find bugs that other tests are unable to; for example,

two classes might work fine on their own, but when used in conjunction, could break each other.

Figure 3. [15] Software Testing Pyramid

In terms of automated testing, unit testing is about as low level as you can go when

testing software. As with the previously mentioned testing methods unit testing can be under

both black box and white box testing depending on how you perform the test.

Figure 4. [18] The diagram above Compares some of the automated testing types.

With unit testing each unit of code is tested independently, you are testing the

functionality of only one piece or unit of code, due to this unit testing is normally performed

during the early stages of software development. Unit tests are generally easy to create and take

very little time to make but only really excel at finding bugs in specific portions or units of code.


7

Benefits and Drawbacks of Using Scripts

Benefits

For a developer to decide whether manual or automated testing should be used, one must

be aware of which type of testing would be best suited for the project. Automated testing allows

for the reusability of scripts by performing multiple and repeated test cases. It allows for parallel

testing on multiple testing platforms, operating systems, and browsers simultaneously, which

improves the coverage of testing as automated execution of test cases significantly cuts down on

testing time. With that being said, the testing script can run at any time without human

intervention. This can reduce business costs and create a better utilization of computer power. As

illustrated in the image below, automation test significantly saves on cost over time [13].

Additionally, automated testing can produce maximum test data to cover many inputs and

expected output for result comparison. Automated testing becomes extremely useful when tests

involve lots of data entry, since manually entering thousands of entries is an inconvenience for

the human tester [6]. Moreover, manual testing is not as reliable as automation testing as it can

be much more prone to error, whereas automated testing can precisely perform the same

operation each time it is run.

Figure 5. [13] Cost and Time Saving Comparison

Drawbacks

However, automation testing also has drawbacks, such that it requires more effort during

the development stage. The testing script must be properly created in order to develop an

effective test script, while manual testing does not require any additional knowledge other than

the functionalities of the application that is being tested. During the test design session, user

interaction, where the developer must create test scripts using test tool features and programming

features, is mandatory, which is more time consuming. More importantly, some testing platform

may have their own defects, so that it may not achieve the desired benefits while performing the

test [7]. Debugging issues can be significant time investment for developers if it is difficult to


8

locate the errors in the script. Also, a poorly designed test script that contains errors, can lead to

further errors within the software being developed.

Although automated testing could reduce business costs, test automation could be more

expensive than manual testing due to a higher license fees to use certain the devices or tools [7].

For example, a company may have to pay to use a software that updates, maintains, and make

improvements to the script. Therefore, it is important for the company to determine what is the

best methodology that will avoid any unnecessary action and additional cost when performing

the testing.

In general, companies and developers try to

automate testing for the maximum number of cases.

However, some tests require human-tester

observation due to environmental limitations, which

cannot automate all testable requirements. For

instance, automated tests are useless in exploratory

testing, which is the process of learning the product,

designing test cases, executing test cases, and

interpreting the test results at the same time. This type

of testing requires the tester to think critically through

the entire process, in which decision making is

essential based on the test scope, charter, and goals

[5]. As you might know, automation has its limits due

to the cognitive skills required for this type of test.

As illustrated in the top portion of the pyramid,

testing such as UX, GUI, and API are difficult or

impossible to be done by running a test program [14].

Attempts to automate those tests will experience

challenges such as complex workflows, time-

consuming processes, and brittle tests. In the web

application, UX, UI, and API are continuously updated.

Thus, maintenance of the automated script for those

tests has been a challenge for a long time. Moreover,

error handling has been an issue with automation

testing since it is extremely difficult to revoke the error

messages. Although it is possible to implement

effective automation scripts for these scenarios, it is extremely time-consuming., making these

tests more effective to conduct manually.

Although automating testing can save time, there will always be the need to run some

tests manually. Therefore, it is important for human testers to be able to manage the requirements

of all test cases. Successfully managing both forms of testing is critical to application success.

Figure 7. Figure 7. [14] Automation Test Pyramid

Figure 6. [5] The diagram above illustrates that

process of exploratory testing


9

Frameworks in Automated Testing

In order to maximize the benefits of automated software testing, one should be aware of

the different types of frameworks that are used so that a successful implementation of a testing

script can be ensured. A framework is composed of a series of steps, or guidelines, for creating

an automated testing script. There are many different frameworks to use depending upon the

preferences of the user. A few examples of such a preference would include the: level of

maintainability of the framework, its complexity to use, or the data that is required to use a

particular framework.

One type of framework is known as the linear scripting framework, which is based upon

using a recording of the input given by the user and replaying that input in future script runs.

This technique is ideal for smaller scripts and is user-friendly for individuals who are not

familiar with automated testing techniques. Also, linear scripting is one of the fastest frameworks

to set up if there is a time constraint. However, since each testing script is created specifically for

a certain input, this technique does not allow for reusability of the test script for other inputs or

programs [1]. A framework better suited for larger scripts would be the modular testing

framework, which is based upon splitting the original script into smaller subsections to test each

section at a time. By splitting the script into subsections, the testing script also becomes easier to

maintain as well. However, this method is not a very reusable technique either when it is being

applied to different sections of the original script or for different scripts as the testing script

would have to be adjusted as the data needed changes.

Unlike linear scripting and modular testing, a data-driven framework is reusable as data

changes. This type of framework is based upon separating the data from the logic of the program,

such as using a text file or Microsoft Excel spreadsheet for the data. Since it is reusable, the data

is not hard coded into the testing script so that the script will work on interchangeable data.

However, it is much more complex to set up this type of framework than linear or module types

so that a more-experienced tester would be best-off utilizing this method [2]. A similar testing

framework is called the keyword-driven framework, which uses a separated file of data to create

a table of the different functions of the script and each of these functions is referred to by a

particular keyword. This technique allows for the testing script to be reusable as well [3]. While

the process of creating the keyword-driven framework is not as difficult, it is supposed to be a

more time-consuming process than the data-driven framework. However, the hybrid-driven

testing automation is used most often in real-world testing as it is a customized variation of any

of the frameworks discussed above. Thus, by following one of the mentioned frameworks, a

successful automated testing script can be created.

Figure 8. [4] The diagram above highlights some of the key aspects of choosing particular a framework. The linear scripting

framework fits into the first box as well for being easy to maintain.


10

Essential Tools and Functions

Frameworks are just one of the tools that are necessary for automating scripting

languages as they provide the guidelines for creating the testing scripts, which results in

improved reusability and maintainability of the code. There is also a tool called the continuous

integration tool (CIT), which allows the user to get feedback about the current script whenever a

change is made. The CIT acts as an automated debugger for every update of the script. For

example, GitHub recently adopted a CIT feature. Also, a bug-tracking feature is another type of

integration tool and it is ideal for automated testing as the bugs that occur are constantly being

tracked so that the issue can be fixed as soon as possible [9]. Another essential tool is the test

management tool, which allows for documentation of the purpose of each different test. There

are instances when not every test case can be trialed using only automated reasoning so that

manual testing must be used. Thus, the test management tool allows for enhanced organization

when utilizing automated testing.

Figure 9. [12] The process of how GitHub is used as a contiguous integration tool in automated testing is shown above, as every

commit to a repository informs the developers if an error has occurred.


11

Application Example

The html script, as well as Scripted Cascading style sheet, produce a submit page which

asks for the client’s user id and password or gives an option to register if he or she is a new user.

These details are then inserted into the mySQL database using a php. Php helps the develop

maintain the database and keeps track of the already signed up users.

The final output of this script is a library application register page.


12

Experiments

Kaitlyn Forsberg

While there has been no prior experiment on scripting language, throughout the research of

scripting language, many new knowledges were gained. As it has already been discussed the

reasoning as to why automated scripting should be used, it is also important to discuss an

example of how it can be used as well. For example, an automated testing script was created for

a function that would return zero whenever zero was multiplied with another number [10]. This

function is also known as a unit test as it is a program that is designed to perform a specific

function.

The automated test was implemented by utilized Java and the JUnit framework, where JUnit is

based upon regression testing. Regression testing is used to ensure that software continues to

work properly after adjustments are made to the program so that it does not become more

difficult to find bugs as a program becomes larger [11]. By creating this automated testing script,

the user was able to test the function more widely to ensure it was working properly in all input

test cases.

Qi Zheng

Some experiments that was done on scripting languages uses the script to automate program

execution on different inputs with large sizes and the record the execution time and write a script

to test the program functionalities.

One of the projects that for

Introduction to Algorithm class

was to run different types of

sorting algorithms with different

input sizes. Due to the amount and

size of inputs that runs on the

program, shell script is used to run

the program and record the data

result, allowing time to work on

something else while the program

executes. Below is a shell script

that used to run all .sh file and

save all output result to a .txt file.

Besides shell script, the scripting

language, Python, is also used to ensure that the program is implemented correctly by conducting

the unit test. Automate scripting is very effective because script testing allows the tester to create

the test cases while the developer is implementing the program, which reduces the time of entire

program development and increases the accuracy of the test result.


13

Elijah Richardson

Experiments utilizing a testing framework called NUnit, optimized for .NET and ReSharper an

application that allows for easier unit test automation. In order to run tests each test is given an

attribute to signify that it’s a test, there are quite a few different types of test attribute you can

give it. [Test], and [TestCase] are the most common and easy to understand attributes.

[TestFixture] is also often seen and just denotes that a class contains unit tests.

In the example a test is being performed to

check whether or not a function that

checks if a number is prime is returning

the correct output. If the function evaluates

a non-prime number as prime the test

catches this and returns “1 should not be

prime”. This was just a simple example

with the value being tested being written

directly into the test, but you can run

multiple tests that testing different

numbers using the [TestCase] attribute.

This can also be combined with ReSharper

to allow for more randomized input and

better test automation.

Muhammad Siddiqui

This python script is written to decode the url into a format that is understood by a user, base64

and python url library are used to achieve the latter. The picture also illustrates the linux shell

where the python is being interpreted by using the base_URL_decider.py. Note that in the print

command there are several base64 commands used, this is to ensure that if the message is longer

than 64 bytes, we decode the string in chunks of 64 bytes until end of file is reached. A better

practice would be to use a while loop with end of file as our delimiter/terminator.

Figure 10. [20] NUnit Test Example


14

The decoded message from the encrypted url key:

b'flag{a11_y0ur_b4s3_R_b3l0ng_2_Us}'

Robinson Simon

Experiments utilizing a library in python called "unittest” were performed to automate testing.

The unittest library allows for an abundance of tools that are available for building and running

tests. An example shown below:

Figure 11. [19] Unit Test Example

In this example, three separate tests are created with methods names beginning with “test” to

signify that it is a unit test. Within each function we can test as many items as necessary. It is

best to only test what the title of the method suggests for easier maintenance. For example, the

second method “test_isupper(self)” has two tests inside. Both tests perform the isupper() function

to show that it works correctly. Utilizing this framework has made unit testing simple and

straightforward to help ensure that the software works as intended.


15

References

[1] Mohamed, et al. “Types of Test Automation Frameworks.” Software Testing Material, 30

Apr. 2020, www.softwaretestingmaterial.com/types-test-automation-frameworks/.

[2] “Test Automation Frameworks - Why, Types, Benefits, Approach.” TestingXperts, 28 Sept.

2020, www.testingxperts.com/blog/test-automation-frameworks.

[3] “Test Automation Frameworks.” Smartbear.com, smartbear.com/learn/automated-

testing/test-automation-frameworks/.

[4] Different Types of Test Automation Frameworks Explained - QARA Enterprise Blog,

www.qaratest.com/blogs/different-types-of-test-automation-frameworks-explained.

[5] “What Is Exploratory Testing? Techniques with Examples.” Guru99,

www.guru99.com/exploratory-testing.html.

[6] “Automating Software Testing - How to Decide.” Perforce Software,

www.perforce.com/blog/alm/which-tests-do-i-automate.

[7] Reddy, G C. “Disadvantages of Test Automation.” Software Testing, 16 June 2018,

www.gcreddy.com/2016/05/disadvantages-of-test-automation.html.

[8] Testim. “What Is Test Automation? A Simple, Clear Introduction.” Fast Authoring of AI-

Stabilized End-to-End Tests-Codeless, Coded, or Both, Fast Authoring of AI-Stabilized End-

to-End Tests-Codeless, Coded, or Both, 3 Feb. 2020, www.testim.io/blog/what-is-test-

automation/.

[9] “Essential Elements of Test Automation.” Testsigma Blog, 5 Oct. 2020,

testsigma.com/blog/essential-elements-of-test-automation/

[10] Lars Vogel (c) 2007 - 2020 vogella GmbH. “Get More...” Vogella.com,

www.vogella.com/tutorials/JUnit/article.html.

[11] “Regression Testing.” Wikipedia, Wikimedia Foundation, 3 Oct. 2020,

en.wikipedia.org/wiki/Regression_testing.

[12] “Continuous Integration with Jenkins And GitHub.” QA Touch, 19 Dec. 2018,

www.qatouch.com/blog/continuous-integration-with-jenkins-and-github/.

[13] “Myths of Test Automation.” QodeStack, 24 Feb. 2019, qodestack.com/myths-of-test-

automation/.

[14] Kyryk, Ivanna. “What Projects Need Test Automation.” QATestLab Blog, 12 June 2018,

blog.qatestlab.com/2018/06/12/when-automate-testing/.

[15] “Maximize Test Automation ROI with Test Data Generation.” GenRocket Blog, 4 Apr.

2019, www.genrocket.com/blog/maximize-test-automation-roi-with-test-data-generation/.

[16] Praveen Mishra ·, et al. “A Guide to Test Automation Types, Tools, and Benefits - DZone

DevOps.” Dzone.com, 20 Sept. 2019, dzone.com/articles/a-guide-to-test-automation-types-

tools-and-benefits.

[17] “7 Different Types of White Box Testing Techniques: White Box Testing Tools.”

TestingGenez, 11 June 2020, testinggenez.com/types-of-white-box-testing-techniques/.

[18] Stf. “Unit Testing.” SOFTWARE TESTING Fundamentals, 13 Sept. 2020,

softwaretestingfundamentals.com/unit-testing/.

[19] “25.3. Unittest - Unit Testing Framework¶.” 25.3. Unittest - Unit Testing Framework -

Python 2.7.18 Documentation, docs.python.org/2/library/unittest.html.

[20] Rprouse. “Unit Testing C# with NUnit and .NET Core - .NET Core.” .NET Core | Microsoft

Docs, docs.microsoft.com/en-us/dotnet/core/testing/unit-testing-with-nunit.

http://www.softwaretestingmaterial.com/types-test-automation-frameworks/

http://www.testingxperts.com/blog/test-automation-frameworks

http://www.qaratest.com/blogs/different-types-of-test-automation-frameworks-explained

http://www.guru99.com/exploratory-testing.html

http://www.vogella.com/tutorials/JUnit/article.html

http://www.qatouch.com/blog/continuous-integration-with-jenkins-and-github/

http://www.genrocket.com/blog/maximize-test-automation-roi-with-test-data-generation/

automated testing using scriptingthe data can then be taken from each area to make improvements....

Documents