Automated Cross-app testing

William Ricardo Ravelo Mendez

May, 2019Version: 1.0

Bogotá, Colombia

Systems and Computing Engineering DepartmentFaculty of Engineering

The Software Design Lab

Undergraduate Thesis

Automated Cross-app testing

William Ricardo Ravelo Mendez

Advisor Ph.D. Mario Linares VásquezSystems and Computing Engineering DepartmentUniversidad de los Andes, Bogotá, Colombia

May, 2019

William Ricardo Ravelo Mendez

Automated Cross-app testing

Undergraduate Thesis, May, 2019

Advisor: Ph.D. Mario Linares Vásquez

Universidad de los Andes

The Software Design Lab

Faculty of Engineering

Systems and Computing Engineering Department

Cra 1 # 18A - 12

111711 and Bogotá, Colombia

Abstract

The number of mobile applications that involve the interaction of two or moreusers are becoming more common each year and the demand of good performanceand availability by their users is increasing. This makes testing and automation ofprocesses essential for delivering high-quality cross-device apps. In this paper, wepresent Kraken, a cross-device testing tool that enables a tester to write, run, andvalidate test scenarios that involve the interaction of more than two devices as wellas its underlying implementation. The proposed tool uses APKs as input along withtests scripts wrote using the Gherkin syntax. Kraken is publicly available at GitHubhttps://bit.ly/2Xjd4Gq. Video: https://bit.ly/2x0qE2A

v

Contents

1 Introduction 11.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.3 Expected results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.4 Obtained results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 Related work 32.1 Mobile end-to-end frameworks . . . . . . . . . . . . . . . . . . . . . 32.2 Octopus and signaling . . . . . . . . . . . . . . . . . . . . . . . . . . 3

3 Kraken 53.1 Proposed approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53.2 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

3.2.1 Signaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63.2.2 writeSignal(channel, content) . . . . . . . . . . . . . . . . . . 73.2.3 readSignal(channel, content, timeout) . . . . . . . . . . . . . . 73.2.4 Scenarios execution . . . . . . . . . . . . . . . . . . . . . . . 8

3.3 Kraken on action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93.3.1 Console Interface . . . . . . . . . . . . . . . . . . . . . . . . . 93.3.2 Calabash steps . . . . . . . . . . . . . . . . . . . . . . . . . . 123.3.3 Test file examples . . . . . . . . . . . . . . . . . . . . . . . . . 143.3.4 Web Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

4 Conclusions 194.1 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194.2 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Bibliography 21

vii

1Introduction

Testing mobile applications has become one of the most important concerns ofsoftware developers in order to validate that its expected behavior is correct; techcompanies relies on testing to ensure quality attributes such as availability, perfor-mance, security, usability, etc. Following the ideal software testing pyramid [9], E2Eautomated tests should be included in the test strategy, this type of testing consists ofvalidating the different flows of an application from start to end. This means runningtest cases which involve different functionalities imitating how a real user wouldinteract with an application. Primarily this type of testing is used for acceptance testsand system tests over the GUI before pushing changes to production with the help ofdifferent frameworks that offer the possibility of testing an app in a black or whitebox manner [4]. At a high level, when talking about mobile white box testing weare referring to the kind of testing where it is required to have access to the sourcecode. Contrarily when talking of mobile black box testing, we refer to the kind oftesting were the source code is not necessary but all test scenarios can be run usingonly the APK (Android package file format).

1.1 Motivation

Even though E2E testing offers the possibility of testing a big variety of flowsand scenarios, a concern is build when the flow under test involves the inter-communication of two or more actors where their behavior can impact the resultof each test case. Consider for example a messaging app, a simple test case wouldinvolve a first user entering the application, adding a user, sending a message,waiting for a response, receiving the response and closing the chat. We would alsorequire to test in parallel the case where the second user in another device opensthe application, receives the message, opens the message and responds to it. Thiscould go even further when its needed to test group chats were the interaction ofthree or more users is involved. Cases like the one mentioned before are becomingmore and more common between applications in the market nowadays [7] (e.g.WhatsApp, Messenger, Uber, Lyft). This is when traditional E2E testing falls shortand the necessity of testing applications in an integration level is required.

1

1.2 Objectives

The main goal of the project is to build an open source E2E testing tool that sup-ports and validates scenarios involving inter-communication between two or moreapplications.

• Build a framework that given a test case can run a series of scenarios in two ormore devices or emulators in parallel.

• Generate an execution report providing information such as device meta-data, number of scenarios failed, number of scenarios passed, execution time,executed steps, execution tree graph and screenshots.

• Validate the framework’s functionalities with real mobile applications thatinvolve intercommunication scenarios.

1.3 Expected results

After finishing the project, it is expected to have a functioning mobile testing toolcapable of running scenarios that depend on the interaction between at least twodevices. This tool should use an existing UI automation framework to write andrun test cases as well as generate execution reports of each scenario ran by thetester. Finally, the framework needs to be published as an open source project andtransformed in a package dependency of a programming language.

1.4 Obtained results

A testing framework called Kraken was build using Ruby as primary programminglanguage and published as a gem in RubyGems.org the biggest package managerfor Ruby. Kraken with the help of Calabash-Android and Cucumber supports andvalidates scenarios that involve the inter-communication between two or moreAndroid applications in a black box manner. The framework also generates executionreports for a tester including relevant data about each test case such as number ofscenarios failed, number of scenarios passed, device metadata, execution time and asankey diagram representing the flow of each test scenario.

2 Chapter 1 Introduction

2Related work

2.1 Mobile end-to-end frameworks

Currently for testing simple flows in mobile applications there is a big variety of E2Eframeworks such as Calabash [5], UIAutomator [3], Espresso [2], Robotium [8]among others. Each one of them presents different benefits that can adjust to theneeds of each tester, for example Calabash offers the possibility of testing in a blackbox manner while using behavior driven testing (BDT) [6], which means that eachtest scenario is written using a simple understandable language that in most cases isplain English, making it easier to understand and write by people that don’t have abroad knowledge in programming. This is achieved using Cucumber and Gherkinsyntax that translates each word or sentence to a block of code. Another example isEspresso that offers tools such as the Espresso Test Recorder where the tester caninteract with the GUI and each interaction is then saved in a generated test casefile.

Even though these frameworks offer many capabilities, currently there is not an offi-cial way of testing scenarios that involve the interaction of two or more applicationsrunning simultaneously. Some of the work arounds that have been proposed is tocommunicate devices via an existing communication protocol such as Bluetooth ifthe devices are close enough, Server Sockets if both devices have internet connectionor even NFC. These approaches introduce new limitations such as no support foroffline-mode scenarios if using internet as the communication protocol, as well aslow consistency because communication technologies can work differently betweendevices and platforms.

2.2 Octopus and signaling

Around the year 2015 the engineering team of Uber faced the problem of test casesthat involved the coordination of two devices, specifically they needed to test thescenario where a passenger requested a ride, a driver accepted this request, traveledfor some time and then finished the route. Even though this may seem as a simplescenario, it requires the communication of two different apps (rider and driver app)

3

running in different devices in order to complete the ride. Facing this problemmotivated Uber to propose a solution named Octopus [1].

Octopus uses a communication protocol called signaling based in the idea that eachemulator or real device has a communication channel where he can receive signalssent from other devices. A signal is a file that contains a string describing an actionthat has been or needs to be executed by a device. This enables a device to stop andwait for a signal whenever it requires an action to be executed before continuing thetest case or to notify another device when an action has been completed.

This framework allows the devices to have a virtual two-way communication viaa test host (Computer running test cases and connected via USB to the devices)responsible of orchestrating signals. Despite Octopus proposing a solution to theinter-communication testing problem, it does not offer the possibility of testing in ablack box manner, making it impossible to test an application directly from its APK.Another problem with this framework is that it’s for internal use only, that meansthat only Uber can use it.

4 Chapter 2 Related work

3Kraken

Kraken is an open source automated android E2E testing tool that supports andvalidates scenarios that involve inter-communication between two or more users. Itworks in a Black Box manner and uses signaling protocol for coordinating the commu-nication between devices. Kraken is publicly available at GitHub https://bit.ly/2Xjd4Gqand as a ruby gem with the package name kraken-mobile.

3.1 Proposed approach

The solution proposed is to extend Calabash-Android UI automation framework torun in more than one device at the same time, this involves modifying its underlyingrunner to start the required number of devices, create the necessary artifacts for eachdevice, support and orchestrate a file based signaling protocol which enables eachdevice to communicate with each other and finally generate an execution report withinformation such as device metadata, number of scenarios failed, number of scenariospassed, execution time, executed steps, execution tree graph and screenshots.

Also, the solution needs to extend Cucumber and Gherkin syntax in order to enablethe tester to run and write test files specifying which scenario is going to run in agiven device or emulator as well as the necessary steps that will execute the signalingfunctions.

3.2 Implementation

To allow cross-device interaction-based testing, the Kraken tool is inspired on bothE2E existing frameworks and Uber’s octopus. It was build on top of the specification-by-example capability provided by cucumber/calabash and combine it with thesignaling approach proposed by Uber’s octopus. Kraken includes five additionalcapabilities not supported by existing E2E testing frameworks: (i) send signal, (ii)read signal, (iii) generate random events over full- and part-of-screen, (iv) Krakenmonkey, and (v) define a data pool of sensitive data such as passwords or API keysthat should be used within the test cases.

5

Figure 3.1 presents the architecture of Kraken. It consists of several modules aimedat launching required devices, creating artifacts for each device, supporting/orches-trating the signaling protocol, executing the steps and generating reports.

APK

TestFiles

Command Line Interface

Gherkin StaticAnalyzer

APK Analyzer

RunnerEngine

DeviceManager

Thread PoolGUI Automation

FrameworkReporterEngine

Signal Engine

Pool of Physical or Virtual Android Devices

TestFiles APK

TestFile

APK

Test FileGrouper

Kraken Execution Engine

1

3

2

45 6

789

10

Fig. 3.1: The Kraken architecture and workflow.

3.2.1 Signaling

The signaling approach is based on the “publish-and-suscribe” architectural style, inwhich the actors (i.e., devices) wait for signals indicating when to start the executionof a set of steps. To achieve this, Kraken has a central Signal Engine that listensfor signals from devices and sends execution actions to the corresponding devicestimely; it is worth noting that this process is not done serially, which allows forparallel execution of scenarios when required.

The Signal Engine coordinates each device and enables their communication by usinga file-based signaling protocol; each device has a text file in its internal file systemcalled inbox where it can receive signals from other devices. These signals are stringswritten to the inbox file that indicates an action completed in another device. Thelack of a message allows devices to recognize an execution point where a wait shouldbe done. To accomplish this, Kraken offers two main functions: readSignal andwriteSignal.

6 Chapter 3 Kraken

3.2.2 writeSignal(channel, content)

The writeSignal function is in charge of sending a signal to another specified device,to achieve this, it receives the following parameters:

• Channel: String containing the tag of the device that is going to receive thesignal. (e.g. @user1)

• Content: String containing a word describing the action completed by thecurrent device and that represents the signal name (e.g. REQUESTED_RIDE)

Finally, to write the content of the signal into the desired inbox, Kraken translatesthe channel parameter into a device id and uses the Unix echo command with thehelp of ADB to modify the inbox file content.

Code example 3.1: ADB write to file

1 $ adb -s {DEVICE_ID} shell echo {CONTENT} > /sdcard/inbox.txt

3.2.3 readSignal(channel, content, timeout)

The readSignal function is in charge of waiting for a signal to be written in thecurrent device inbox, to achieve this it receives the following parameters:

• Channel: String containing the tag of the device that is going to receive thesignal. (e.g @user1)

• Content: String containing the signal name that is expected to be received.(e.g REQUESTED_RIDE)

• Timeout: Integer representing the number of seconds the device is going towait for a signal.

Finally, to wait for a signal in the desired inbox, Kraken translates the channelparameter into a device id and runs a loop that every second reads the last line ofits inbox file and compares it to the expected content. When the content matchesthe expected signal, the function returns making it possible for the calabash-androidprocess to continue, but if the content does not match and the execution time reachesthe timeout, then the process raises an exception assuming the test as failed.

3.2 Implementation 7

For reading the last line of the inbox, Kraken uses the Unix cat command with thehelp of ADB.

Code example 3.2: ADB read from file

1 $ adb -s {DEVICE_ID} shell cat /sdcard/inbox.txt

3.2.4 Scenarios execution

To execute the scenarios a user can upload an app .apk file and all its test files writtenusing the Gherkin syntax with the help of the Command Line Interface. Once theuser has provided these files, Kraken validates: (i) the provided APK is signed withthe Kraken user’s keystore (using the APK Analyzer) and (ii) via the Gherkin StaticAnalyzer component, it checks the test files are in compliance with the Gherkin andKraken syntaxes.

To start the testing process the Runner Engine will first check the number of availabledevices or emulators and with the help of the Test File Grouper component, Krakenmaps each test file with a device where it is going to be executed. Next, the RunnerEngine will notify de Device Manager to start the testing process in the requireddevices or emulators, each one of them running on a different thread ensuring anisolated environment; also the Device Manager will create an inbox file in each devicewhere the signals are going to be received. Every test process will start thanks to theGUI Automation Framework component in charge of interpreting every instructionspecified in the test files, executing the instruction on a device, asserting that theaction is completed correctly and finally saving a report.

To implement the signaling actions, when the GUI Automation Framework encountersone of these instructions in the test file it pauses its execution and notifies the SignalEngine. When it is required to send a signal from one device to another, this enginewill write the signal content to the destination inbox file and will inform the GUIAutomation Framework that the action has been completed in order to continue theprocessing of instructions.

On the other hand, when a device needs to wait for a signal, the Signal Enginecreates a loop that continuously reads the device inbox file and returns when theexpected signal content is received or the default timeout is reached. Finally, whenall instructions have been processed and completed, the GUI Automation Frameworknotifies the Reporter Engine component in order to read the logs and results saved,and generates an HTML report.

8 Chapter 3 Kraken

3.3 Kraken on action

To install Kraken as a Ruby gem a tester can run the following command.

Code example 3.3: Kraken install command

1 $ gem install kraken-mobile

3.3.1 Console Interface

User interaction with Kraken is done via console commands. Once the Kraken gemis installed, users can (i) generate a test folder skeleton, (ii) list the devices attachedto the computer, (iii) setup the user tag for each device (i.e., the way how eachdevice is identified in the Kraken framework), (iv) sign an apk with the currentlyconfigured keystore, and (v) run a test. The first step to use Kraken is generating thebase project that contains a feature .feature file (similar to the one in Code example3.4), a subfolder containing a ruby file for the step_definitions, and a subfolderwith support files required for Kraken execution; to this, the user should call thekraken-mobile gen command.

Code example 3.4: Example feature file

1 Feature: Example feature2

3 @user14 Scenario: As a first user I say hi to a second user5 Given I wait6 Then I send a signal to user 2 containing "hi"7

8 @user29 Scenario: As a second user I wait for user 1 to say hi

10 Given I wait for a signal containing "hi"11 Then I wait

A feature file should contain scenarios definition following a Gherkin+Kraken syntax.For example, the feature in Code example 3.4 contains two scenarios, one per eachdevice involved in the test. However, as it can be seen in lines 3 and 8 of Codeexample 3.4, each scenario is linked to an specific user (i.e., a device). The usertag follows a naming pattern: @user(\d+). The tags are automatically assigned byKraken to each device connected to the execution engine; the tags can be checkeda-priori by using the kraken-mobile devices command, which presents the devicesinformation (including the tags). An example of the command output is presentedin Code example 3.5.

Code example 3.5: Kraken devices command output

1 $ kraken-mobile devices2 List of devices attached

3.3 Kraken on action 9

3 user1 - emulator-5554 - Android_SDK_built_for_x864 user2 - emulator-5556 - Android_SDK_built_for_x86

If an user wants to modify the setup parameters for a test (i.e., amount of devices,tag assigned to each device and apk to be used in each device), he can do it via thekraken-mobile setup command. Examples of the output can be seen in the followingsnippet:

1 $ kraken-mobile setup2 How many devices do you want to use? 23 Choose your user 1 emulator-55544 What APK user 1 is going to run? ~/Desktop/app.apk5 Choose your user 1 emulator-55566 What APK user 1 is going to run? ~/Desktop/app.apk7 Saved your settings to .kraken_mobile_settings. You can edit the settings manually or

run this setup script again

Due to the usage of calabash as part of the architecture, Kraken requires an APKunder test to be signed with the same keystore as the one currently configured; it isdone by using the kraken-mobile resign <apkPath> command.

Once the setup is done, the user must modify the .feature file provided by Kraken,with her specific requirements. It is worth remembering that Kraken uses the Gherkinsyntax, thus, the tester can use any of the steps already defined in Calabash[5]. TheKraken specific steps are:

Send signal

As it was mentioned previously, Kraken provides a step that sends a signal to a device.This step has two parameters: the user (i.e., tag) that will receive the signal and thecontent of the signal. The structure of this step is:

Code example 3.6: Kraken send signal

1 I send a signal to user (\d+) containing "([^\"]*)"|

Read signal

This step can be used in three different ways: first, to define only the timeout toreceive a signal (See line 1 in Code example 3.7); second, to set the expected content(see line 2 in Code example 3.7); and third, to define the expected content andtimeout (see line 3 in Code example 3.7)

10 Chapter 3 Kraken

Code example 3.7: Kraken send signal

1 I wait for any signal for (\d+) seconds2 I wait for a signal containing "([^\"]*)"3 I wait for a signal containing "([^\"]*)" for (\d+) seconds

Random Events Scenario

Kraken provides a scenario step that allows users to generate GUI-based randominputs by following the syntax presented in Code example 3.9. This step can executea given number random events over the full screen (line 1 in Code example 3.9) oron a region specified by the user as a portion of the screen (line 2 in Code example3.9).

Code example 3.8: Kraken random step

1 I start a monkey with (\d+) events2 I start a monkey with (\d+) events from height (\d+)

Kraken Monkey

Kraken introduces another step that generalizes the execution of random events, byalso sending and reading signals randomly. The structure of this step is:

Code example 3.9: Kraken random step

1 I start kraken monkey with (\d+) events

Properties file

Kraken uses properties files to store sensitive data such as passwords or API keysthat should be used in your test cases. The properties files should be a manuallycreated JSON file with the structure presented in the following snippet:

1 {2 "@user1": {3 "PASSWORD": "test"4 }5 }

Note that the key-value pairs are organized by user tags. In order to use the valuesdefined in the properties file, the property should be invoked using “< ... >” as inthe following: I see the text "<PASSWORD>".

3.3 Kraken on action 11

Finally, after writing the scenarios, the last step for using Kraken is running it withthe kraken-mobile run <apk> command; it must be called in the folder that contains the.features files. However, as it was mentioned before Kraken can be used with adifferent APK for each device. Therefore, when a setup has been done, the commandmust include the flag: --configuration=<conf_file_path>. In case the user wants touse a property file, then the command must include --properties=<properties_path

>.

3.3.2 Calabash steps

In this section we will list some of the most common predefined steps offered byCalabash.

Assertion

To assert that a text can be found in some of the elements visible in the currentactivity XML the following steps can be used.

Code example 3.10: Positive assertion steps

1 Then I see the text "([^\"]*)"2 Then I see "([^\"]*)"3 Then I should see "([^\"]*)"4 Then I should see text containing "([^\"]*)"

To assert that a text can not be found in some of the elements visible in the currentactivity XML the following steps can be used.

Code example 3.11: Negative assertion steps

1 Then I should not see "([^\"]*)"2 Then I don’t see the text "([^\"]*)"3 Then I don’t see "([^\"]*)"

Input

To enter text in one of the inputs visible in the current activity XML the tester canuse one the following steps.

Code example 3.12: Input steps

1 Then I enter "([^\"]*)" into input field number (\d+)2 Then I enter text "([^\"]*)" into field with id "([^\"]*)"

12 Chapter 3 Kraken

Touching

To press a button or a specific view visible in the current activity XML calabash offersthe following steps.

Code example 3.13: Touching steps

1 Given I press the "([^\"]*)" button2 Then I press button number (\d+)3 Then I press view with id "([^\"]*)"4 Then I press "([^\"]*)"5 Then I touch the "([^\"]*)" text

Gestures

To interact with the screen using gestures the tester can use the following steps.

Code example 3.14: Gesture steps

1 Then I swipe left2 Then I swipe right3 Then I scroll down4 Then I scroll up

Waiting

To stop the execution and wait for a specific number of seconds or for a view toappear, the tester can use the following steps

Code example 3.15: Waiting steps

1 Then I wait up to (\d+) seconds for "([^\"]*)" to appear2 Then I wait up to (\d+) seconds to see "([^\"]*)"3 Then I wait for "([^\"]*)" to appear4 Then I wait to see "([^\"]*)"5 Then I wait for (\d+) seconds

Screenshots

To take a screenshot of the current activity, calabash offers the following steps.

Code example 3.16: Screenshot steps

1 Then take picture2 Then I take a picture

3.3 Kraken on action 13

3 Then I take a screenshot

3.3.3 Test file examples

The following section presents two different examples of test files written usingthe tool. Source code, more examples and videos showing Kraken on action areavailable at https://bit.ly/2Xjd4Gq.

Spotify and Shazam example

This scenario tests Shazam functionality of listening to a song and recognizing it’sname. To achieve this, one device plays a song using Spotify APK while in anotherdevice Shazam listens to the song.

Code example 3.17: Shazam and Spotify feature file

1 Feature: Shazam and Spotify feature2

3 @user24 Scenario: As a user I listen with Shazam to a song and recognize it.5 Given I wait for a signal containing "ready_to_play" for 60 seconds6 Then I click on screen 50% from the left and 50% from the top7 Then I send a signal to user 1 containing "start_playing"8 Then I wait for "<SONG_NAME>" to appear9 Then I see the text "<SONG_NAME>"

10 Then I see the text "<BAND_NAME>"11 Then I send a signal to user 1 containing "stop_playing"12

13 @user114 Scenario: As a valid user I can log into spotify and play a song.15 Given I wait16 Then I press view with id "button_login"17 Then I press view with id "username_text"18 Then I enter email "<EMAIL>"19 Then I hide my keyboard20 Then I press view with id "password_text"21 Then I enter password "<PASSWORD>"22 Then I hide my keyboard23 Then I press view with id "login_button"24 Then I press view with id "search_tab"25 Then I press view with id "find_search_field"26 Then I press view with id "query"27 Then I enter song name "<SONG_NAME>"28 Then I hide my keyboard29 Then I wait for "<SONG_NAME>" to appear30 Then I send a signal to user 2 containing "ready_to_play"31 Then I wait for a signal containing "start_playing" for 30 seconds32 Then I press view with id "text1"33 Then I wait for a signal containing "stop_playing" for 30 seconds34 Then I press view with id "playPause"

14 Chapter 3 Kraken

QuizUp example

QuizUp is an Android trivia app that allows a user to invite and compete with friends.This scenario test the functionality of inviting a friend and playing both at the sametime in order to answer a list of questions.

Code example 3.18: QuizUp feature file

1 Feature: QuizUp feature2

3 @user14 Scenario: As a first user I wait for a friend invitation and answer list of questions5 Given I wait6 Then I click on screen 0% from the left and 95% from the top7 Then I wait8 Then I touch the "Log In with Email" text9 Then I wait

10 Then I enter email11 Then I enter password12 Then I press view with id "email_login_button"13 Then I send a signal to user 2 containing "finished_login_user1"14 Then I wait for a signal containing "finished_login_user2" for 60 seconds15 Then I wait for a signal containing "sended_invitation" for 60 seconds16 Then I touch the "ACCEPT" text17 Then I send a signal to user 2 containing "accepted_invitation"18 Then I answer a question19 Then I answer a question20 Then I answer a question21 Then I answer a question22 Then I answer a question23 Then I answer a question24 Then I answer a question25

26 @user227 Scenario: As a first user I invite a friend to answer a list of questions28 Given I wait29 Then I click on screen 0% from the left and 100% from the top30 Then I wait31 Then I touch the "Log In with Email" text32 Then I wait33 Then I enter email 234 Then I enter password 235 Then I press view with id "email_login_button"36 Then I wait for 4 seconds37 Then I touch the "Logos" text38 Then I wait39 Then I touch the "Play" text40 Then I touch the "Test" text41 Then I wait for a signal containing "finished_login_user1" for 60 seconds42 Then I send a signal to user 1 containing "finished_login_user2"43 Then I send a signal to user 1 containing "sended_invitation"44 Then I wait for a signal containing "accepted_invitation" for 60 seconds45 Then I answer a question46 Then I answer a question47 Then I answer a question48 Then I answer a question49 Then I answer a question50 Then I answer a question

3.3 Kraken on action 15

51 Then I answer a question

3.3.4 Web Reports

When finishing the execution of the test files, Kraken will generate a directory (inthe current folder) that contains three reports: (i) general report, (ii) features reportby device, and (iii) feature detail by device.

General report: In the top section of this report Kraken provides metadata of all thedevices used in the execution of the test files. A Sankey diagram is included with thepurpose of showing the actions flow of each device in every feature, as in a timeline(Figure 3.2). In this type of graph, the nodes will represent an action performed bya device; when the viewer hovers over the node it shows the name of the action aswell as a thumbnail showing the GUI state after tge action was completed. If thecolor of the node is green it represents that the action passed, otherwise, if the nodeis red then it indicates an error in the execution. Each path, composed of edges withthe same color, represents a different device.

Fig. 3.2: General report - sankey diagram

Features report (by device): In this report, the tester can see the execution re-sults of the features, given a specific device. It includes the number of featuresfailed/passed, APK path, execution time, device name, and feature name (Figure3.3).

16 Chapter 3 Kraken

Fig. 3.3: Features report by device

Feature detail (by device): This final report presents the results of a specific featurefor a given device. The top section presents the number of steps passed and failed,in addition to the list of steps executed. Finally, in the bottom section of the report,every step is presented with a screenshot of the device after the action was executed(Figure 3.4).

Fig. 3.4: Feature details by device

3.3 Kraken on action 17

4Conclusions

After finishing the development of the framework, all primary objectives werecompleted and the tool was tested and published as an open source project and Rubygem. The following chapter will present the current limitations and future work ofKraken.

4.1 Limitations

• The framework can only run in Android devices, this presents a limitationwhen the scenario involves the coordination with devices running differentoperating systems such as iOS.

• Kraken unlike Calabash-Android does not support running in continuous inte-gration engines, this can increase the time of deployment considering that thetesters need to run the framework locally.

• The effects of a UI change can be more detrimental in comparison to normalE2E frameworks because it can imply that all scenarios for every device mustchange. This means that the effort increases in a factor of N where N is thenumber of devices.

• Even though Kraken does not have a maximum number of devices that can runa test, this number is limited to the CPU and RAM the machine of the testerhas.

4.2 Future work

The framework presents many opportunities of improvement some of those inspiredin the current limitations:

• Allow the execution of test scenarios in iOS devices.

• Support at least one continuous integration engine.

19

• Ripper that creates his model based in the snapshot of two or more devices ata given time.

• Fake data generator for text input steps.

• Different implementations of the signaling protocol such as publish and sub-scribe.

20 Chapter 4 Conclusions

Bibliography

[1] B. Jiang A. Chow. Octopus to the Rescue: The Fascinating World of Inter-AppCommunications at Uber Engineering. 2015. URL: https://eng.uber.com/rescued-by-octopus/ (cit. on p. 4).

[2] Android. Espresso. 2019. URL: https://developer.android.com/training/testing/espresso (cit. on p. 3).

[3] Android. UI Automator. 2019. URL: https : / / developer . android . com /training/testing/ui-automator (cit. on p. 3).

[4] Peter Babington. “Software Testing”. In: 1st ed. Jones and Bartlett Learning,2009, pp. 2–3 (cit. on p. 1).

[5] Calabash. Calabash-Android. 2019. URL: http://calaba.sh/ (cit. on pp. 3,10).

[6] A. Hellesy M. Wynne. “The Cucumber for Java book”. In: 1st ed. PragmaticBookshelf, 2012, pp. 4–5 (cit. on p. 3).

[7] Susan Moore. Gartner Says Demand for Enterprise Mobile Apps Will Out- stripAvailable Development Capacity Five to One. 2015. URL: https://www.gartner.com/en/newsroom/press-releases/2015-06-16-gartner-says-demand-for-enterprise-mobile-apps-will-outstrip-available-development-capacity-five-to-one (cit. on p. 1).

[8] RobotiumTech. Robotium. 2019. URL: https://github.com/RobotiumTech/robotium (cit. on p. 3).

[9] Ham Vocke. The Practical Test Pyramid. 2018. URL: https://martinfowler.com/articles/practical-test-pyramid.html (cit. on p. 1).

21

List of Figures

3.1 The Kraken architecture and workflow. . . . . . . . . . . . . . . . . . . 63.2 General report - sankey diagram . . . . . . . . . . . . . . . . . . . . . . 163.3 Features report by device . . . . . . . . . . . . . . . . . . . . . . . . . . 173.4 Feature details by device . . . . . . . . . . . . . . . . . . . . . . . . . . 17

23

Colophon

This thesis was typeset with LATEX 2ε. It uses the Clean Thesis style developed byRicardo Langner.