using gui ripping for automated testing of android apps
DESCRIPTION
As mobile applications become more complex and business-critical, use of well-defined software engineering techniques becomes essential to assure the necessary software quality. In particular, testing and its automation play a strategic part for assuring the quality of applications that are often developed by small teams, working on strict timelines and under the pressure of short time-to-market. This paper presents an automated GUI based testing technique for Android apps. The technique is based on a ripper that automatically explores the GUI with the aim of exercising the application and revealing run-time crashes. At the same time, the ripper builds a GUI model and an executable test suite based on the JUnit test framework. The technique has been evaluated by an experiment where the ripper has been used to test a real Android application. The experiment has shown the fault-detection capability of the technique and its cost-effectiveness in smoke testing processes.TRANSCRIPT
Using GUI Ripping for Using GUI Ripping for Automated Testing of Android Automated Testing of Android
AppsApps
Domenico AmalfitanoDomenico Amalfitano
Salvatore De CarmineSalvatore De Carmine
Anna Rita FasolinoAnna Rita Fasolino
Porfirio Tramontana Porfirio Tramontana
Dipartimento di Informatica e SistemisticaDipartimento di Informatica e Sistemistica
University of Naples Federico II, ItalyUniversity of Naples Federico II, Italy
Atif MemonAtif MemonDepartment of Computer ScienceDepartment of Computer Science
University of MarylandUniversity of Maryland
Porfirio Tramontana – ASE 2012 – Essen – September 6, Porfirio Tramontana – ASE 2012 – Essen – September 6, 20122012
22
MotivationMotivation Android is a very popular operative system and framework for Android is a very popular operative system and framework for
mobile devicesmobile devices Its diffusion is increasing: recently it overtook ioS and Symbian in terms of Its diffusion is increasing: recently it overtook ioS and Symbian in terms of
diffusiondiffusion Over than 500,000 apps are now on the Google Play marketOver than 500,000 apps are now on the Google Play market The most part of the apps are realized by single programmers or little teams The most part of the apps are realized by single programmers or little teams
and lacks in qualityand lacks in quality
Android apps are often developed according to an evolutionary Android apps are often developed according to an evolutionary life cycle, with very frequent releases via Google Playlife cycle, with very frequent releases via Google Play Bugs causing failures and, in particular, crashes are very commonBugs causing failures and, in particular, crashes are very common
There is a strong need for There is a strong need for testing automation testing automation processes processes and and toolstools In this talk, our techniques and tool supporting a process of Android In this talk, our techniques and tool supporting a process of Android
GUI Testing Automation will be presentedGUI Testing Automation will be presented
Porfirio Tramontana – ASE 2012 – Essen – September 6, Porfirio Tramontana – ASE 2012 – Essen – September 6, 20122012
Android appsAndroid apps
An Android application is composed of several types of Java An Android application is composed of several types of Java components instantiated at run-time:components instantiated at run-time:
Activities, Services, Intents, Broadcast Receivers and Content Providers. Activities, Services, Intents, Broadcast Receivers and Content Providers.
An An ActivityActivity component: component: is responsible for presenting a visual user interface; is responsible for presenting a visual user interface; provides a screen with which the users can interact in order to provides a screen with which the users can interact in order to
do something.do something. An Android application usually includes one or more Activity An Android application usually includes one or more Activity
classes that extend the base Activity class provided by the classes that extend the base Activity class provided by the Android development framework.Android development framework.
Android Applications are Android Applications are Event-Driven SystemsEvent-Driven Systems Three types of events can be triggered:Three types of events can be triggered:
user events (such as Click, MouseOver, etc.);user events (such as Click, MouseOver, etc.); events related to the lifecycle of the Activity classesevents related to the lifecycle of the Activity classes
onStart, on Pause, onResume, onStop, onDestroy, …onStart, on Pause, onResume, onStop, onDestroy, … events that are triggered by other input sourcesevents that are triggered by other input sources
such as GPS receiver, phone, network, etc.such as GPS receiver, phone, network, etc.
33
Porfirio Tramontana – ASE 2012 – Essen – September 6, Porfirio Tramontana – ASE 2012 – Essen – September 6, 20122012
Existing approaches to Android Testing Existing approaches to Android Testing AutomationAutomation
JUnit TestingJUnit Testing Android Testing FrameworkAndroid Testing Framework
Supports the coding of executable test classesSupports the coding of executable test classes Classes supporting the instrumentation and monitoring of Android apps are provided in the Classes supporting the instrumentation and monitoring of Android apps are provided in the
Android framework itselfAndroid framework itself Libraries such as Libraries such as Robotium Robotium and and MonkeyrunnerMonkeyrunner extends the framework extends the framework
Random TestingRandom Testing MonkeyMonkey (included in the Android framework) supports the randomly (included in the Android framework) supports the randomly
automatic exploration of Android appsautomatic exploration of Android apps
Model Based TestingModel Based Testing TEMA ToolsTEMA Tools (Takala, Katara, Harty, 2011) represent a first tentative of (Takala, Katara, Harty, 2011) represent a first tentative of
approach for the generation of test code from a design model of approach for the generation of test code from a design model of Android appAndroid app
44
Porfirio Tramontana – ASE 2012 – Essen – September 6, Porfirio Tramontana – ASE 2012 – Essen – September 6, 20122012
Android GUI RippingAndroid GUI Ripping
We propose a technique and a tool for:We propose a technique and a tool for: Automatic exploration of the GUI of an Automatic exploration of the GUI of an
Android application via a systematic Android application via a systematic RippingRipping approachapproach
Similar to approaches adopted for Web crawlingSimilar to approaches adopted for Web crawling Crash testing Crash testing GUI model reverse engineeringGUI model reverse engineering JUnit test case generationJUnit test case generation
55
Porfirio Tramontana – ASE 2012 – Essen – September 6, Porfirio Tramontana – ASE 2012 – Essen – September 6, 20122012
Android GUI Ripping AlgorithmAndroid GUI Ripping Algorithm
CustomizableCustomizable Selection of types of events to be fired Selection of types of events to be fired Adoption of a GUI traversing strategy (e.g. depth first, breadth Adoption of a GUI traversing strategy (e.g. depth first, breadth
first, etc.)first, etc.) Termination criteria used to stop the GUI traversalTermination criteria used to stop the GUI traversal Setting of specific preconditionsSetting of specific preconditions
Efficient Efficient Provides heuristic solutions to the well known problem of GUI Provides heuristic solutions to the well known problem of GUI
interface states and events explosioninterface states and events explosion GUI exploration criteria based on a Interface Equivalence GUI exploration criteria based on a Interface Equivalence
CriterionCriterion
66
Porfirio Tramontana – ASE 2012 – Essen – September 6, Porfirio Tramontana – ASE 2012 – Essen – September 6, 20122012
1. Task List Initialization;2. while (Task List Is Not Empty) {3. Extract a Task From The Task List;4. Execute the Task;5. Abstract the Current GUI Abstract
State;6. Update the GUI Tree Model;7. if (GUI Exploration Criterion) then
{8. Define New Tasks ;9. Add New Tasks To The Task
List;10. }
11. }
77
RotateRotate
Press MenuPress MenuClick RefreshClick RefreshClick Click New New PostPost
Click PagesClick Pages
Click AboutClick AboutClick Add Click Add AccountAccount
≡≡ ≡≡
Click Click EditEdit
CrashCrash
……
……
Click SaveClick Save
…………
Ripping Ripping AlgorithmAlgorithm
1. Task List Initialization;2. while (Task List Is Not
Empty) {3. Extract a Task From
The Task List;4. Execute the Task;5. Abstract the Current
GUI Abstract State;6. Update the GUI Tree
Model;7. if (GUI Exploration
Criterion) then {8. Define New Tasks ;9. Add New Tasks To
The Task List;10. }11. }
Porfirio Tramontana – ASE 2012 – Essen – September 6, Porfirio Tramontana – ASE 2012 – Essen – September 6, 20122012
Crash Testing ProcessCrash Testing Process
Preconconditions are Preconconditions are set by generating a set by generating a Snapshot Image of the Snapshot Image of the device for the emulatordevice for the emulator
In the task execution In the task execution step, the GUI Ripping step, the GUI Ripping algorithm is executedalgorithm is executed
88
Porfirio Tramontana – ASE 2012 – Essen – September 6, Porfirio Tramontana – ASE 2012 – Essen – September 6, 20122012
A Demonstrational exampleA Demonstrational example
Android for WordpressAndroid for Wordpress A Wordpress client for A Wordpress client for
AndroidAndroid 334 classes, ~10,000 LOCs334 classes, ~10,000 LOCs More than 250,000 downloadsMore than 250,000 downloads
A set of three preconditionsA set of three preconditions No LoginNo Login Single BlogSingle Blog Multiple BlogsMultiple Blogs
Comparison between Comparison between Ripping and Random Testing Ripping and Random Testing with Monkeywith Monkey
99
Measured metricsMeasured metrics EffectivenessEffectiveness
Number of crashes Number of crashes foundfound
Number of Number of different bugs different bugs causing crashescausing crashes
LOCs coverageLOCs coverage
EfficiencyEfficiency Number of test Number of test
casescases Number of eventsNumber of events Ripping time Ripping time
Porfirio Tramontana – ASE 2012 – Essen – September 6, Porfirio Tramontana – ASE 2012 – Essen – September 6, 20122012
ResultsResults
1010
Preconditions
No Login Single Blog Multiple Blog
Random (Single Blog)
#Crashes 0 6 8 3
#Bugs 0 3 4 1
LOC Cov % 2.65 39.32 37.83 25.27
Time (h) 0.2 4.88 4.58 4.46
#TC 18 166 166 -
#Events 36 517 502 14400
Porfirio Tramontana – ASE 2012 – Essen – September 6, Porfirio Tramontana – ASE 2012 – Essen – September 6, 20122012
Bugs and CrashesBugs and Crashes
Id Crash Description Java Exception
Ticket and Changeset
B1 The app crashes trying to opening the default post “Hello World”
StringIndexOutOfBoundException
https://android.trac.wordpress.org/ticket/206https://android.trac.wordpress.org/changeset/398
B2 The app crashes when the Stats activity is rapidly opened and closed (via the Back key).
BadTokenException
https://android.trac.wordpress.org/ticket/208https://android.trac.wordpress.org/changeset/420
B3 The app crashes when the Stats activity is open and the Refresh button is clicked while the progress bar widget is still loading.
NullPointerException
https://android.trac.wordpress.org/ticket/212
https://android.trac.wordpress.org/changeset/423
B4 The app crashes when the user opens a post and tries to share it within his blog. The crash occurs when there is a single blog in the app.
NullPointerException
https://android.trac.wordpress.org/ticket/218https://android.trac.wordpress.org/changeset/446
1111
The 14 crashes The 14 crashes found by the found by the Ripping process Ripping process belongs to four belongs to four different bugsdifferent bugs Bugs have been Bugs have been
submitted to the submitted to the bug tracking bug tracking system and system and accepted/fixed by accepted/fixed by the programmerthe programmer
Monkey was able Monkey was able to find only one of to find only one of these bugsthese bugs
Porfirio Tramontana – ASE 2012 – Essen – September 6, Porfirio Tramontana – ASE 2012 – Essen – September 6, 20122012
1212
Conclusions and Future WorksConclusions and Future Works
Our technique supports the execution of Our technique supports the execution of completely automatic generation and completely automatic generation and execution of test cases for Android execution of test cases for Android applications. applications. In our preliminary demonstration example:In our preliminary demonstration example: Real bugs and crashes have been detected;Real bugs and crashes have been detected; Good level of code coverage has been reached;Good level of code coverage has been reached; Testing time is compatible with smoke testing Testing time is compatible with smoke testing
processes;processes; Reusable test cases has been automatically Reusable test cases has been automatically
generated;generated; Our technique demonstrates its effectiveness with Our technique demonstrates its effectiveness with
respect to standard Random testing tools such as respect to standard Random testing tools such as MonkeyMonkey
Porfirio Tramontana – ASE 2012 – Essen – September 6, Porfirio Tramontana – ASE 2012 – Essen – September 6, 20122012
1313
Future WorksFuture Works
This presentation covers the first part of a This presentation covers the first part of a larger project on testing automation of Android larger project on testing automation of Android applicationsapplications
Some future worksSome future works Publication of the tool Publication of the tool
As a standalone toolAs a standalone tool As a serviceAs a service
A preliminary demo is available at A preliminary demo is available at http://wpage.unina.it/ptramont/GUIRipperWiki.htmhttp://wpage.unina.it/ptramont/GUIRipperWiki.htm
Further experiments with more applications and Further experiments with more applications and ripping parametersripping parameters
Delay Time, ripping strategy (depth first/breadth first, Delay Time, ripping strategy (depth first/breadth first, activity equivalence criteria, maximum depth), input activity equivalence criteria, maximum depth), input generation strategies, …generation strategies, …
Sensor event testingSensor event testing
Porfirio Tramontana – ASE 2012 – Essen – September 6, Porfirio Tramontana – ASE 2012 – Essen – September 6, 20122012
1414