Automated Reporting of GUI Design Violations for Mobile AppsKevin Moran Boyang Li Carlos Bernal-Caacuterdenas Dan Jelf and Denys Poshyvanyk

College of William amp MaryDepartment of Computer Science

Williamsburg VA USAkpmoranboyangcebernaldkjelfdenyscswmedu

ABSTRACTThe inception of a mobile app often takes form of a mock-up ofthe Graphical User Interface (GUI) represented as a static imagedelineating the proper layout and style of GUI widgets that satisfyrequirements Following this initial mock-up the design artifactsare then handed off to developers whose goal is to accurately im-plement these GUIs and the desired functionality in code Giventhe sizable abstraction gap between mock-ups and code developersoften introduce mistakes related to the GUI that can negativelyimpact an apprsquos success in highly competitive marketplaces More-over such mistakes are common in the evolutionary context ofrapidly changing apps This leads to the time-consuming and labo-rious task of design teams verifying that each screen of an app wasimplemented according to intended design specifications

This paper introduces a novel automated approach for verifyingwhether the GUI of a mobile app was implemented according to itsintended design Our approach resolves GUI-related informationfrom both implemented apps and mock-ups and uses computervision techniques to identify common errors in the implementationsof mobile GUIs We implemented this approach for Android in a toolcalled Gvt and carried out both a controlled empirical evaluationwith open-source apps as well as an industrial evaluation withdesigners and developers from Huawei The results show that Gvtsolves an important difficult and highly practical problem withremarkable efficiency and accuracy and is both useful and scalablefrom the point of view of industrial designers and developers Thetool is currently used by over one-thousand industrial designers ampdevelopers at Huawei to improve the quality of their mobile apps

CCS CONCEPTSbull Software and its engineering rarr Software design engineer-ing Requirements analysis

ACM Reference FormatKevin Moran Boyang Li Carlos Bernal-Caacuterdenas Dan Jelf and DenysPoshyvanyk 2018 Automated Reporting of GUI Design Violations for Mo-bile Apps In ICSE rsquo18 ICSE rsquo18 40th International Conference on SoftwareEngineering May 27-June 3 2018 Gothenburg Sweden ACM New YorkNY USA 11 pages httpsdoiorg10114531801553180246

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page Copyrights for components of this work owned by others than theauthor(s) must be honored Abstracting with credit is permitted To copy otherwise orrepublish to post on servers or to redistribute to lists requires prior specific permissionandor a fee Request permissions from permissionsacmorgICSE rsquo18 May 27-June 3 2018 Gothenburg Swedencopy 2018 Copyright held by the ownerauthor(s) Publication rights licensed to theAssociation for Computing MachineryACM ISBN 978-1-4503-5638-11805 $1500httpsdoiorg10114531801553180246

1 INTRODUCTIONIntuitive elegant graphical user interfaces (GUIs) embodying effec-tive user experience (UX) and user interface (UI) design principlesare essential to the success of mobile apps In fact one may arguethat these design principles are largely responsible for launchingthe modern mobile platforms that have become so popular todayApple Incrsquos launch of the iPhone in 2007 revolutionized the mobilehandset industry (heavily influencing derivative platforms includ-ing Android) and largely centered on an elegant well-thought outUX experience putting multitouch gestures and a natural GUI atthe forefront of the platform experience A decade later the mostsuccessful mobile apps on todayrsquos highly competitive app stores(eg Google Play[5] and Applersquos App Store[3]) are those that em-brace this focus on ease of use and blend intuitive user experienceswith beautiful interfaces In fact given the high number of apps intodayrsquos marketplaces that perform remarkably similar functions [7]the design and user experience of an app are often differentiatingfactors leading to either success or failure [12]

Given the importance of a proper user interface and user expe-rience for mobile apps development usually begins with UIUXdesign experts creating highly detailed mock-ups of app screensusing one of several different prototyping techniques [25 44] Themost popular of these techniques and the focus of this paper isreferred to as mock-up driven development where a designer (orgroup of designers) creates pixel perfect representations of appUIs using software such as Sketch[10] or PhotoShop[1] Once thedesign artifacts (or mock-ups) are completed they are handed offto development teams who are responsible for implementing thedesigns in code for a target platform In order for the design envi-sioned by the UIUX experts (who carry domain knowledge thatfront-end developers may lack) to be properly transferred to usersan accurate translation of the mock-up to code is essential

Yet implementing an intuitive and visually appealing UI in codeis well-known to be a challenging undertaking [37 39 46] As suchmanymobile development platforms such as Applersquos Xcode IDE andAndroid Studio include powerful built-in GUI editors Despite theease of use such technologies are intended to facilitate a controlledstudy has illustrated that such interface builders can be difficult tooperate with users prone to introducing bugs [49] Because appsunder development are prone to errors in their GUIs this typicallyresults in an iterative workflow where UIUX teams will frequentlymanually audit app implementations during the development cycleand report any violations to the engineering team who then aimsto fix them This incredibly time consuming back-and-forth processis further complicated by several underlying challenges specificto mobile app development including (i) continuous pressure forfrequent releases [22 24] (ii) the need to address user reviews

arX

iv1

802

0473

2v2

[cs

SE

] 5

Apr

201

8

ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk

quickly to improve app quality [19 20 40 41] (iii) frequent platformupdates and API instability [15 27 28 33] including changes inUIUX design paradigms inducing the need for GUI re-designs(eg material design) and (iv) the need for custom componentsand layouts to support complex design mock-ups Thus there isa practical need for effective automated support to improve theprocess of detecting and reporting design violations and providingdevelopers with more accurate and actionable information

The difficulty that developers experience in creating effectiveGUIs stems from the need to manually bridge a staggering abstrac-tion gap that involves reasoning concise and accurate UI code frompixel-based graphical representations of GUIs The GUI errors thatare introduced when attempting to bridge this gap are known inliterature as presentation failures Presentation failures have beendefined in the context of web applications in previous work as ldquoadiscrepancy between the actual appearance of a webpage [or mobileapp screen] and its intended appearance [32] We take previousinnovative work that aims to detect presentation errors in web ap-plications [18 31 32 42] as motivation to design equally effectiveapproaches in the domain of mobile apps Presentation failuresare typically comprised of several visual symptoms or specific mis-matches between visual facets of the intended GUI design and theimplementation of those GUI-components [32] in an app Thesevisual symptoms can vary in type and frequency depending on thedomain (eg web vs mobile) and in the context of mock-up drivendevelopment we define them as design violations

In this paper we present an approach calledGvt (GuiVerificationsysTem) developed in close collaboration with Huawei Our ap-proach is capable of automated precise reporting of the designviolations that induce presentation failures between an app mock-up and its implementation Our technique decodes the hierarchalstructure present in both mockups and dynamic representationsof app GUIs effectively matching the corresponding componentsGvt then uses a combination of computer vision techniques toaccurately detect design violations Finally Gvt constructs a reportcontaining screenshots links to static code information (if code isprovided ) and precise descriptions of design violations GVT wasdeveloped to be practical and scalable was built in close collaborationwith the UIUX teams at Huawei and is currently in use by overone-thousand designers and engineers at the company

To evaluate the performance and usefulness ofGvtwe conductedthree complementary studies First we empirically validated Gvtrsquosperformance by measuring the precision and recall of detectingsynthetically injected design violations in popular open sourceapps Second we conducted a user study to measure the usefulnessof our tool comparing Gvtrsquos ability to detect and report designviolations to the ability of developers while also measuring theperceived utility of Gvt reports Finally to measure the applicabil-ity of our approach in an industrial context we present the resultsof an industrial case study including (i) findings from a surveysent to industrial developers and designers who use Gvt in theirdevelopment workflow and (ii) semi-structured interviews withboth design and development team managers about the impact ofthe tool Our findings from this wide-ranging evaluation includethe following key points (i) In our study using synthetic violationsGvt is able to detect design violations with an overall precisionof 98 and recall of 96 (ii) Gvt is able to outperform developers

with Android development experience in identifying design viola-tions while taking less time (iii) developers generally found Gvtrsquosreports useful for quickly identifying different types of design vio-lations and (iv) Gvt had a meaningful impact on the design anddevelopment of mobile apps for our industrial partner contributingto increased UIUX quality

Our paper contributions can be summarized as followsbull We formalize the concepts of presentation failures and designviolations for mock-up driven development in the domain ofmobile apps and empirically derive common types of designviolations in a study on an industrial dataset

bull We present a novel approach for detecting and reportingthese violations embodied in a tool called Gvt that uses hier-archal representations of an apprsquos GUI and computer visiontechniques to detect and accurately report design violations

bull We conduct a wide-ranging evaluation of the Gvt studyingits performance usefulness and industrial applicability

bull We include an online appendix [35] with examples of reportsgenerated by Gvt and our evaluation dataset Additionallywe make the Gvt tool and code available upon request

2 PROBLEM STATEMENT amp ORIGINIn this section we formalize the problem of detecting design viola-tions in GUIs of mobile apps and discuss the origin of the problemrooted in industrial mobile app design amp development21 Problem StatementAt a high level our goal is to develop an automated approachcapable of detecting classifying and accurately describing designviolations that exist for a single screen of a mobile app to helpdevelopers resolve presentation failures more effectively In thissection we formalize this scenario in order to allow for an accuratedescription and scope of our proposed approach While this sectionfocuses on concepts Sec 4 focuses on the implementation details

211 GUI-Components amp Screens There are two main logi-cal constructs that define the concept of the GUI of an app GUI-components (or GUI-widgets) and Screens A GUI-component is adiscrete object with a set of attributes (such as size and locationamong others) associated with a particular Screen of an app AScreen is an invisible canvas of size corresponding to the physi-cal screen dimensions of a mobile device We define two types ofscreens those created by designers using professional-grade toolslike Sketch and those collected from implemented apps at runtimeEach of these two types of Screens has an associated set of GUI-components (or components) Each set of components associatedwith a screen is structured as a cumulative hierarchy comprising atree structure starting with a single root node where the spatiallayout of parent always encompasses contained child componentsDefinition 1 GUI-Component (GC) - A discrete object GC witha corresponding set of attributes a which can be represented asa four-tuple in the form (ltx-positiony-positiongt ltheightwidthgtlttextgt ltimagegt) Here the first four elements of the tuple describethe location of the top left point for the bounding box of the com-ponent and the height and width attributes describe the size of thebounding box The text attribute corresponds to text displayed bythe component and the image attribute represents an image of thecomponent with bounds adhering to the first two attributes

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden

A) App Mock-Up B) App Implementation

GUI-Component

Screen

Presentation Failurewith three DesignViolations (Color Size Location)

Presentation Failurewith one DesignViolation (Font)

Figure 1 Examples of Formal Definitions

Definition 2 Screen (S) - A canvas S with a predefined heightand width corresponding to the physical display dimensions of asmartphone or tablet Each Screen contains a cumulative hierarchyof components which can be represented as a nested set such that

S = GC1GC2GCi GC3 (1)

where each GC has a unique attribute tuple and the nested set canbe ordered in either depth-first (Exp 1) or in a breadth-first mannerWe are concerned with two specific types of screens screens repre-senting mock-ups of mobile apps Sm and screens representing realimplementations of these apps or Sr

212 Design Violations amp Presentation Failures As describedearlier design violations correspond to visual symptoms of pre-sentation failures or differences between the intended design andimplementation of a mobile app screen Presentation failures canbe made up of one or more design violations of different typesDefinition 3 Design Violation (DV) - As shown in Exp 2 amismatch between the attribute tuples of two corresponding leaf-level (ie having no direct children) GUI-componentsGCmi andGCrjof two screens Sm and Sr imply a design violation DV associatedwith those components In this definition leaf nodes correspond toone another if their location and size on a screen (ie ltx-positiony-positiongt ltheightwidthgt) match within a given threshold Equalitybetween leaf nodes is measured as a tighter matching thresholdacross all attributes As we illustrate in the next section inequalitiesbetween different attributes in the associated tuples of the GCs leadto different types of design violations

(GCmi asymp GCrj ) and (GCmi GCrj )

=rArr DV isin GCmi GCrj

(2)

Definition 4 Presentation Failure (PF) - A set of one or moreDVs attributed to a set of corresponding GUI-components betweentwo screens Sm and Sr as shown in Exp 3 For instance as shownin Fig 1 a single set of corresponding components may have dif-ferences in both the ltxygt and ltheightwidthgt attributes leading totwo constituent design violations that induce a single presentationfailure PF Thus each presentation failure between two Screens Scorresponds to at least one mismatch between the attribute vectorsof two corresponding leaf node GUI-components GCim and GCir

if DV1DV2 DVi isin GCmi GCrj

then PF isin Sm Sr (3)

213 Problem Statement Given these definitions the problembeing solved in this paper is the following Given two screens Smand Sr corresponding to the mock-up and implementation screensof a mobile application we aim to detect and describe the set ofpresentation failures PF1 PF2 PFi isin Sm Sr Thus we aimto report all design violations on corresponding GC pairs

DV1DV2 DVk isinGCmi1 GC

rj1 GC

mi2 GC

rj2 GC

mix GC

rjy

(4)

22 Industrial Problem OriginsA typical industrial mobile development process includes the fol-lowing steps (as confirmed by our collaborators at Huawei) (i)First a team of designers creates highly detailed mockups of anapprsquos screens using the Sketch [10] (or similar) prototyping soft-ware These mock-ups are typically ldquopixel-perfect representationsof the app for a given screen dimension (ii) The mock-ups arethen handed off to developers in the form of exported images withdesigner added annotations stipulating spatial information andconstraints Developers use this information to implement repre-sentations of the GUIs for Android using a combination of Javaand xml (iii) Next after the initial version of the app has beenimplemented compiled Android Application Package(s) (ie apks)are sent back to the designers who then install these apps on targetdevices generate screenshots for the screens in question and man-ually search for discrepancies compared to the original mock-ups(iv) Once the set of violations are identified these are communi-cated back to the developers via textual descriptions and annotatedscreenshots at the cost of significant manual effort from the designteams Developers must then identify and resolve the DVs usingthis information The process is often repeated in several iterationscausing substantial delays in the development process

The goal of our work is to drastically improve this iterativeprocess by (i) automating the identification of DVs on the screensof mobile apps - saving both the design and development teamstime and effort and (ii) providing highly accurate information tothe developers regarding these DVs in the form of detailed reports -in order to reduce their effort in resolving the problem

3 DESIGN VIOLATIONS IN PRACTICEIn order to gain a better understanding of the types ofDVs that occurin mobile apps in practice we conducted a study using a datasetfrom Huawei While there do exist a small collection of taxonomiesrelated to visual GUI defects [23 26] and faults in mobile apps [2129] we chose to conduct a contextualized study with our industrialpartner for the following reasons (i) existing taxonomies for visualGUI defects were not detailed enough containing only generalfaults (eg ldquoincorrect appearancerdquo) (ii) existing fault taxonomiesfor mobile apps either did not contain visual GUI faults or werenot complete and (iii) we wanted to derive a contextualized DVtaxonomy for apps developed at Huawei The findings from thisstudy underscore the existence and importance of the problem thatour approach aims to solve in this context Due to an NDA we arenot able to share the dataset or highlight specific examples in orderto avoid revealing information about future products at HuaweiHowever we present aggregate results in this section

ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk

Layout Violations

39

Text Violations

23

Resource Violations

38

Horizontal Translation

10

Vertical Translation

16HampVSize5

H Size4

HampVTrans

5

Font Style Violation

8

Font Color Violation

9

Incorrect Text Violation

6

Image Color Violation

10

Missing Component12

Extra Component

6

Incorrect Image

7

Component Shape 1 Component Type 1

H = Horizontal V = VerticalTrans = Translation

Figure 2 Distribution of Different Types of Industrial DVs

31 Study Setting amp MethodologyThe goal of this study is to derive a taxonomy of the differenttypes of DVs and examine the distribution of these types inducedduring the mobile app development process The context of thisstudy is comprised of a set of 71 representative mobile app mock-up and implementation screen pairs from more than 12 differentinternal apps annotated by design teams from our industrial partnerto highlight specific instances of resolved DVs This set of screenpairs was specifically selected by the industrial design team tobe representative both in terms of diversity and distribution ofviolations that typically occur during the development process

In order to develop a taxonomy and distribution of the violationspresent in this dataset we implement an open coding methodologyconsistent with constructivist grounded theory [17] Following theadvice of recent work within the SE community [45] we stipulateour specific implementation of this type of grounded theory whilediscussing our deviations from the methods in the literature Wederived our implementation from the material discussed in [17]involving the following steps (i) establishing a research problemand questions (ii) data-collection and initial coding and (iii) focusedcoding We excluded other steps described in [17] such as memoingbecause we were building a taxonomy of labels and seeking newspecific data due to our NDA limiting the data that could be sharedThe study addressed the following research question What are thedifferent types and distributions of GUI design violations that occurduring industrial mobile app development processes

During the initial coding process three of the authors were sentthe full set of 71 screen pairs and were asked to code four pieces ofinformation for each example (i) a general category for the viola-tion (ii) a specific description of the violation (iii) the severity ofthe violation (if applicable) and (iv) the Android GC types affected(eg button) Finally we performed a second round of coding thatcombined the concepts of focused and axial coding as described in[17] During this round two of the authors merged the responsesfrom all three types of coding information where at least two ofthe three coders agreed During this phase similar coding labels

were merged (eg ldquolayout violation vs ldquospatial violation) conflictswere resolved two screen pairs were discarded due to ambiguityand cohesive categories and subcategories were formed The authoragreement for each of the four types of tags is as follows (i) generalviolation category (100) (ii) specific violation description (96)(iii) violation severity (100) and (iv) affected GC types (845)32 Grounded Theory Study ResultsOur study revealed three major categories of design violationseach with several specific subtypes We forgo detailed descriptionsand examples of violations due to space limitations but provideexamples in our online appendix [35] The derived categories andsubcategories of DVs and their distributions are illustrated in Fig 2Overall 82 DVs were identified across the 71 unique screen pairsconsidered in our study The most prevalent category of DVs in ourtaxonomy are Layout Violations (asymp 40) which concern either atranslation of a component in the x or y direction or a change in thecomponent size with translations being more common The secondmost prevalent category (asymp 36) consists of Resource Violationswhich concern missing components extra components color dif-ferences and image differences Finally about one-quarter (asymp 24)of these violations are Text Violations which concern differences incomponents that display text We observed that violations typicallyonly surfaced for ldquoleaf-level components in the GUI hierarchyThat is violations typically only affected atomic components ampnot containers or backgrounds Only 582 of examined violations(asymp 6) affected backgrounds or containers Even in these few casesthe violations also affected ldquoleaf-level components

The different types of violations correspond to different inequali-ties between the attribute tuples of corresponding GUI-componentsdefined in Sec 2 This taxonomy shows that designers are chargedwith identifying several different types of design violations a daunt-ing task particularly for hundreds of screens across several apps

4 THE GVT APPROACH41 Approach OverviewThe workflow of Gvt (Fig 3) proceeds in three stages First in theGUI-Collection Stage GUI-related information from both mock-upsand running apps is collected Next in theGUI-Comprehension Stageleaf-level GCs are parsed from the trees and a KNN-based algorithmis used to match corresponding GCs using spatial information Fi-nally in the Design Violation Detection Stage DVs are detected usinga combination of methods that leverage spatial GC information andcomputer vision techniques42 Stage 1 GUI Collection

421 Mock-Up GUI Collection Software UIUX design profes-sionals typically use professional-grade image editing software(such as Photoshop[1] or Sketch[10]) to create their mock-ups De-signers employed by our industrial partner utilize the Sketch designsoftware Sketch is popular among mobile UIUX designers due toits simple but powerful features ease of use and large library ofextensions [11] When using these tools designers often constructgraphical representations of smartphone applications by placingobjects representing GCs (which we refer to as mock-up GCs) on acanvas (representing a Screen S) that matches the typical displaysize of a target device In order to capture information encodedin these mock-ups we decided to leverage an export format that

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden3 Design Violation Detection

2 GUI-Comprehension11 Mock-Up GUI Collection

12 App Implementation GUI Collection

Resource Violation Detector

Text Violation Detector

GVT GUI Collector

apk

orapp src

Physical Deviceor Emulator

Android UIAutomator

Android adb

Screencap

AndroidApplication

Marketch Javascript

File+

+

Export Screenshot

and Marketch

Files

Mock-Up GUI Hierarchy

Implementation GUI Hierarchy

UIX Parser

GUI Hierarchy

Construction

Detection of Corresponding

Leaf Node Pairs

Identified Corresponding Component Pairs

Difference Image

Layout Violation Detector

Sketch GUI-Parser

Perceptual Image

Differencing (PID)

UIXxml File

Checksltimagegtlttextgt

Color Quantization amp Histogram Analysis

Checksltimagegt

+Missing amp Extraneous

Components

Color Quantization amp Histogram Analysis

Javascript Parser

GUIHierarchy

Construction

UIX Parser

Violation Manager

Checksltwidthgtltheighgt

Checksltxgtltygt

Design Violation Resolver

Difference Image Parser

MismatchedComponentPairs

ne

Spatial Comparator

Size Comparator

Component-LevelPerceptual Image Diff

Figure 3 Overview of GVTWorkflowwas already in use by our industrial partner an open-source Sketchextension called Marketch [6] that exports mock-ups as an html

page including a screenshot and JavaScript fileThus as input from the mock-up Gvt receives a screenshot (to

be used later in the Design Violation Detection Phase) and a directorycontaining the Marketch information The JavaScript file containsseveral pieces of information for each mock-up GC including (i) thelocation of the mock-up GC on the canvas (ii) size of the boundingbox and (iii) the textfont displayed by the mock-up GC (if any) Asshown in Figure 3- 11 we built a parser to read this informationHowever it should be noted that our approach is not tightly coupled toSketch or Marketch files1 After the Marketch files have been parsedGvt examines the extracted spatial information to build a GC hier-archy The result can be logically represented as a rooted tree whereleaf nodes contain the atomic UI-elements with which a typicaluser might interact Non-leaf node components typically representcontainers that form logical groupings of leaf node componentsand other containers In certain cases our approximation of usingmock-up GCs to represent implementation GCs may not hold Forinstance an icon which should be represented as a single GC mayconsist of several mock-up GCs representing parts of the icon Gvthandles such cases in the GUI-Comprehension stage

422 Dynamic App GUI-Collection In order to compare thethe mock-up of an app to its implementation Gvt must extractGUI-related meta-data from a running Android app Gvt is able touse Androidrsquos uiautomator framework [2] intended for UI testingto capture xml files and screenshots for a target screen of an apprunning on a physical device or emulator Each uiautomator filecontains information related to the runtime GUI-hierarchy of thetarget app including the following attributes utilized by Gvt (i)The Android component type (eg androidwidgetImageButton)(ii) the location on the screen (iii) the size of the bounding box (iv)text displayed (v) a developer assigned id The hierarchal structureof components is encoded directly in the uiautomator file and thuswe built a parser to extract GUI-hierarchy using this informationdirectly (see Fig 3- 12 )

43 Stage 2 GUI ComprehensionIn order for Gvt to find visual discrepancies between componentsexisting in the mock-up and implementation of an app it must1Similar information regarding mock-up GCs can be parsed from the html or ScalableVector Graphics (svg) format exported by other tools such as Photoshop[1]

determine which components correspond to one another Unfor-tunately the GUI-hierarchies parsed from both the Marketch anduiautomator files tend to differ dramatically due to several factorsmaking tree-based GC matching difficult First since the hierarchyconstructed using the Marketch files is generated using informationfrom the Sketch mock-up of app it is using information derivedfrom designers While designers have tremendous expertise in con-structing visual representations of apps they typically do not takethe time to construct programmatically-oriented groupings of com-ponents Furthermore designers are typically not aware of thecorrect Android component types that should be attributed to dif-ferent objects in a mock-up Second the uiautomator representationof the GUI-hierarchy contains the runtime hierarchal structure ofGCs and correct GC types This tree is typically far more com-plex containing several levels of containers grouping GCs togetherwhich is required for the responsive layouts typical of mobile apps

To overcome this challenge Gvt instead forms two collectionsof leaf-node components from both the mock-up and implementa-tion GUI-hierarchies (Fig 3- 2 ) as this information can be easilyextracted As we reported in Sec 3 the vast majority of DVs af-fects leaf-node components Once the leaf node components havebeen extracted from each hierarchy GVT employs a K-Nearest-Neighbors (KNN) algorithm utilizing a similarity function basedon the location and size of the GCs in order to perform matchingIn this setting an input leaf-node component from the mock-upwould be matched against it closest (eg K=1) neighbor from theimplementation based upon the following similarity function

γ = (|xm minus xr | + |ym minus yr | + |wm minuswr | + |hm minus hr |) (5)

Where γ is a similarity score where smaller values represent closermatches The x yw and h variables correspond to the x amp y lo-cation of the top and left-hand borders of the bounding box andthe height and width of the bounding boxes for the mock-up andimplementation GCs respectively The result is a list of GCs thatshould logically correspond to one another (corresponding GCs)

It is possible that there exist instances of missing or extraneouscomponents between the mock-up and implementation To identifythese cases our KNN algorithm employs a GC-Matching Threshold(MT ) If the similarity score of the nearest neighbor match for agiven input mock-up GC exceeds this threshold it is not matchedwith any component and will be reported as amissing GC violation

ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk

If there are unmatched GCs from the implementation they are laterreported as extraneous GC violations

Also there may be cases where a logical GC in the implemen-tation is represented as small group of mock-up GCs Gvt is ableto handle these cases using the similarity function outlined aboveFor each mock-up GC Gvt checks whether the neighboring GCsin the mockup are closer than the closest corresponding GC inthe implementation If this is the case they are merged with theprocess repeating until a logical GUI-component is represented44 Stage 3 Design Violation DetectionIn the Design Violation Detection stage of the Gvt workflow theapproach uses a combination of computer vision techniques andheuristic checking in order to effectively detect the different cate-gories of DVs derived in our taxonomy presented in Section 3

441 Perceptual Image Differencing In order to determine cor-responding GCs with visual discrepancies Gvt uses a techniquecalled Perceptual Image Differencing (PID) [48] that operates uponthe mock-up and implementation screenshots PID utilizes a modelof the human visual system to compare two images and detectvisual differences and has been used to successfully identify visualdiscrepancies in web applications in previous work [31 32] Weuse this algorithm in conjunction with the GC information derivedin the previous steps of Gvt to achieve accurate violation detec-tion For a full description of the algorithm we refer readers to[48] The PID algorithm uses several adjustable parameters includ-ing F which corresponds to the visual field of view in degrees Lwhich indicates the luminance or brightness of the image and Cwhich adjusts sensitivity to color differences The values used inour implementation are stipulated in Section 45

The output of the PID algorithm is a single difference image (Fig3- 3 ) containing difference pixels which are pixels considered to beperceptually different between the two images After processing thedifference image generated by PID Gvt extracts the implementa-tion bounding box for each corresponding pair of GCs and overlaysthe box on top of the generated difference image It then calculatesthe number of difference pixels contained within the bounding boxwhere higher numbers of difference pixels indicate potential visualdiscrepancies Thus Gvt collects all ldquosuspicious GC pairs with a of difference pixels higher than a Difference Threshold DT This setof suspicious components is then passed to the Violation Manager(Fig 3- 3 ) so that specific instances of DVs can be detected

442 Detecting Layout Violations The first general category ofDVs thatGvt detects are Layout Violations According the taxonomyderived in Sec 3 there are six specific layout DV categories thatrelate to two component properties (i) screen location (ie ltxygtposition) and (ii) size (ie lthwgt of the GC bounding box) Gvt firstchecks for the three types of translationDVs utilizing a heuristic thatmeasures the distance from the top and left-hand edges of matchedcomponents If the difference between the components in either thex ory dimension is greater than a Layout Threshold (LT ) then thesecomponents are reported as a Layout DV Using the LT avoids triviallocation discrepancies within design tolerances being reported asviolations and can be set by a designer or developer using the toolWhen detecting the three types of size DVs in the derived designviolation taxonomyGvt utilizes a heuristic that compares thewidthand height of the bounding boxes of corresponding components If

the width or height of the bounding boxes differ by more than theLT then a layout violation is reported

443 Detecting Text Violations The next general type of DVthat Gvt detects are Text Violations of which there are three spe-cific types (i) Font Color (ii) Font Style and (iii) Incorrect TextContent These detection strategies are only applied to pairs oftext-based components as determined by uiautomator informationTo detect font color violations Gvt extracts cropped images foreach pair of suspicious text components by cropping the mock-upand implementation screenshots according to the componentrsquos re-spective bounding boxes Next Color Quantization (CQ) is appliedto accumulate instances of all unique RGB values expressed in thecomponent-specific images This quantization information is thenused to construct a Color Histogram (CH) (Fig 3- 3 ) Gvt computesthe normalized Euclidean distance between the extracted ColorHistograms for the corresponding GC pairs and if the Histogramsdo not match within a Color Threshold (CT) then a Font-Color DVis reported and the top-3 colors (ie centroids) from each CH arerecorded in theGvt report Likewise if the colors domatch then thePID discrepancy identified earlier is due to the Font-Style changing(provided no existing layout DVs) and thus a Font-Style Violationis reported Finally to detect incorrect text content Gvt utilizesthe textual information preprocessed to remove whitespace andnormalize letter cases and performs a string comparison If thestrings do not match then an Incorrect Text Content DV is reported

444 Detecting Resource Violations Gvt is able to detect thefollowing resource DVs (i) missing component (ii) extraneous com-ponent (iii) image color (iv) incorrect images and (v) componentshape The detection and distinction between Incorrect Image DVsand Image Color DVs requires an analysis that combines two differ-ent computer vision techniques To perform this analysis croppedimages from the mock-up and implementation screenshots accord-ing to corresponding GCs respective bounding boxes are extractedThe goal of this analysis is to determine when the content of image-based GCs differ as opposed to only the colors of the GCs differingTo accomplish this Gvt leverages PID applied to extracted GCimages converted to a binary color space (B-PID) in order to detectdifferences in content and CQ and CH analysis to determine differ-ences in color (Sec 443) To perform the B-PID procedure croppedGC images are converted to a binary color space by extracting pixelintensities and then applying a binary transformation to the inten-sity values (eg converting the images to intensity independentblack amp white) Then PID is run on the color-neutral version ofthese images If the images differ by more than an Image DifferenceThreshold (IDT ) then an Incorrect Image DV (which encompassesthe Component Shape DV ) is reported If the component passes thebinary PID check then Gvt utilizes the same CQ and CH process-ing technique described above to detect image color DVs Missingand extraneous components are detected as described in Sec 43

445 Generating Violation Reports In order to provide develop-ers and designers with effective information regarding the detectedDVs Gvt generates an html report that for each detected viola-tion contains the following (i) a natural language description ofthe design violation(s) (ii) an annotated screenshot of the app im-plementation with the affected GUI-component highlighted (iii)cropped screenshots of the affected GCs from both the design and

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden

implementation screenshots (iv) links to affected lines of applica-tion source code (v) color information extracted from the CH forGCs identified to have color mismatches and (vi) the differenceimage generated by PID The source code links are generated bymatching the ids extracted from the uiautomator information backto their declarations in the layout xml files in the source code (egthose located in the res directory of an apprsquos source code) Weprovide examples of generated reports in our online appendix [35]45 Implementation amp Industrial CollaborationOur implementation of Gvt was developed in Java with a SwingGUI In addition to running the Gvt analysis the tool executableallows for one-click capture of uiautomator files and screenshotsfrom a connected device or emulator Several acceptance tests ofmock-upimplementation screen pairs with pre-existing violationsfrom apps under development within our industrial partner wereused to guide the development of the tool 12 Periodic releases ofbinaries for bothWindows and Mac were made to deploy the tool todesigners and developers within the company The authors of thispaper held regular bi-weekly meetings with members of the designand development teams to plan features and collect feedback

Using the acceptance tests and feedback from our collaboratorswe tuned the various thresholds and parameters of the tool for bestperformance The PID algorithm settings were tuned for sensitivityto capture subtle visual inconsistencies which are then later filteredthrough additional CV techniques F was set to 45 L was set to100cdm2 and C was set to 1 The GC-Matching Threshold (MC)was set to 18th the screen width of a target device the DT fordetermining suspicious GCs was set to 20 The LT was set to 5pixels (based on designer preference) the CT which determinesthe degree to which colors must match for color-based DVs was setto 85 and finally the IDT was set to 20 Gvt allows for a userto change these settings if desired additionally users are capableof defining areas of dynamic content (eg loaded from networkactivity) which should be ignored by the Gvt analysis5 DESIGN OF THE EXPERIMENTSTo evaluate Gvtrsquos performance usefulness and applicability weperform three complimentary studies answering the following RQs

bull RQ1 How well does Gvt perform in terms of detecting andclassifying design violations

bull RQ2 What utility can Gvt provide from the viewpoint ofAndroid developers

bull RQ3What is the industrial applicability of Gvt in terms ofimproving the mobile application development workflow

RQ1 and RQ2 focus on quantitatively measuring the performanceof Gvt and the utility it provides to developers through a controlledempirical and a user study respectively RQ3 reports the results ofa survey and semi-structured interviews with our collaboratorsaimed at investigating the industrial applicability of Gvt51 Study 1 Gvt Effectiveness amp PerformanceThe goal of the first study is to quantitatively measure Gvt in termsof its precision and recall in both detecting and classifying DVs

511 Study Context To carry out a controlled quantitativestudy we manually reverse engineered Sketch mockups for tenscreens for eight of the most popular apps on Google Play To de-rive this set we downloaded the top-10 apps from each category on

the Google-Play store removing the various categories correspond-ing to games (as these have non-standard GUI-components thatGvt does not support) We then randomly sampled one app fromeach of the remaining 33 categories eliminating duplicates (sinceapps can belong to more than one category) We then manuallycollected screenshots and uiautomator files from two screens foreach application using a Nexus 5 attempting to capture the ldquomainrdquoscreen that a user would typically interact with and one secondaryscreen Using the uiautomator files we generated cropped screen-shots of all the leaf nodes components for each screen of the appFrom these we were able generate 10 screens from 8 applicationsthat successfully ran through Gvt without any reported violations

512 Synthetic DV Injection With a set of correct mock-ups cor-responding to implementation screens in an app we needed a suit-able method to introduce DVs into our subjects To this end we con-structed a synthetic DV injection tool that modifies the uiautomator

xml files and corresponding screenshots in order to introduce de-sign violations from our taxonomy presented in Sec 3 The toolis composed of two components (i) an XML Parser that reads andextracts components from the screen then (ii) a Violation Generatorthat randomly selects components and injects synthetic violationsWe implemented injection for the following types of DVsLocation Violation The component is moved either horizontallyvertically or in both directions within the same container Howeverthemaximum distance from the original point is limited by a quarterof the width of the screen size This was based on the severity ofLayout Violations in our study described in Section 3 In order togenerate the image we cropped the component and moved it to thenew location replacing all the original pixels by the most prominentcolor from the surroundings in the original locationSize Violation The component size either increases or decreasesby 20 of the original size For instances where the component sizedecreases we replaced all the pixels by the most prominent colorfrom the surroundings of the original sizeMissing Component Violation This violation removes a leafcomponent from the screen replacing the original pixels by themost prominent color from the surrounding backgroundImage Violation We perturb 40 of the pixels in an image byrandomly generating an RGB value for the pixels affectedImage Color Violation This rule perturbs the color of an imageby shifting the hue of image colors by 30degComponent Color Violation This uses the same process as forImage Color Violations but we change the color by 180degFont Violation This violation randomly selects a font from theset of Arial Comic Sans MS Courier Roboto or Times Roman andapplies it to a TextView componentFont Color Violation changes the text color of a TextView com-ponent We extracted the text color using CH analysis then wechanged the color using same strategy as for Image Color Violations

513 Study Methodology In injecting the synthetic faults wetook several measures to simulate the creation of realistic faultsFirst we delineated 200 different types of design violations accord-ing to the distribution defined in our DV taxonomy in Sec 3 Wethen created a pool of 100 screens by creating random copies ofthe both the uiautomator xml files and screenshots from our initialset of 10 screens We then used the synthetic DV injection tool toseed faults into the pool of 100 screens according to the following

ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk

criteria (i) No screen can contain more than 3 injected DVs (ii)each GC should have a maximum of 1 DV injected and (iii) Eachscreen must have at least 1 injected DV After the DVs were seededeach of the 100 screens and 200 DVs were manually inspected forcorrectness Due to the random nature of the tool a small num-ber of erroneous DVs were excluded and regenerated during thisprocess (eg color perturbed to perceptually similar color) Thebreakdown of injected DVs is shown in Figure 4 and the full datasetwith description is included in our online appendix [35]

Once the final set of screens with injected violations was derivedwe ran Gvt across these subjects and measured four metrics (i)detection precision (DP ) (ii) classification precision (CP ) (iii) recall(R) and (iv) execution time per screen (ET ) We make a distinctionbetween detection and classification in our dataset because it is pos-sible thatGvt is capable of detecting but misclassifying a particularDV (eg an image color DV misclassified as an incorrect image DV )DP CP and R were measured according to the following formulas

DP CP =Tp

Tp + FpR =

Tp

Tp + Fn(6)

where for DP Tp represent injected design violations that weredetected and for CP Tp represents injected violations that wereboth detected and classified correctly In each case Fp correspondto detected DVs that were either not injected or misclassified ForRecallTp represents injected violations that were correctly detectedand Fn represents injected violations that were not detected Tocollect these measures two authors manually examined the reportsfrom Gvt in order to collect the metrics

52 Study 2 Gvt UtilitySince the ultimate goal of an approach like Gvt is to improvethe workflow of developers the goal of this second study is tomeasure the utility (ie benefit) that Gvt provides to developersby investigating two phenomena (i) The accuracy and effort ofdevelopers in detecting and classifying DVs and (ii) the perceivedutility of Gvt reports in helping to identify and resolve DVs

521 Study Context We randomly derived two sets of screensto investigate the two phenomena outlined above First we ran-domly sampled two mutually exclusive sets of 25 and 20 screensrespectively from the 100 used in Study 1 ensuring at least oneinstance of each type of DV was included in the set This resulted inboth sets of screens containing 40 design violations in total The cor-rect mockup screenshot corresponding to each screen sampled fromthe study were also extracted creating pairs of ldquocorrect mockupand ldquoincorrect implementation screenshots 10 participants withat least 5 years of Android development experience were contactedvia email to participate in the survey

522 Study Methodology We created an online survey withfour sections In the first section participants were given back-ground information regarding the definition of DVs and the differ-ent types of DVs derived in our taxonomy In the second sectionparticipants were asked about demographic information such asprogramming experience and education level In the third sectioneach participant was exposed to 5 mock-up implementation screenpairs (displayed side by side on the survey web page) and asked toidentify any observed design violations Descriptions of the DVswere given at the top of this page for reference For each screenpair participants were presented with a dropdown menu to select

80

85

90

95

100

Font

Colo

r(18)

Font

Styl

e(18

)

Imag

e Colo

r(22)

Imag

e(16

)

Miss

ing G

C(26)

Size(1

8)

Text

Conte

nt(1

4)

Trans

lation

(68)

DV Type

Per

cent

age

MetricCPDPR

Figure 4 Study 1 - Detection Precision (DP ) ClassificationPrecision (CP ) and Recall (R)a type for an observed DV and a text field to describe the errorin more detail For each participant one of the 5 mock-up screenswas a control containing no injected violations The 25 screenswere assigned to participants such that each screen was observedby two participants and the order of the screens presented to eachparticipant was randomized to avoid bias To measure the effective-ness of participants in detecting and describing DVs we leveragethe DP CP and R metrics introduced in Study 1 In the fourth sec-tion participants were presented with two screen pairs from thesecond set of 20 sampled from the user study as well as the Gvtreports for these screens Participants were then asked to answer5 user-preferences (UP) and 5 user experience (UX) questions aboutthese reports which are presented in the following section The UPquestions were developed according to the user experience hon-eycomb originally developed by Morville [36] and were posed toparticipants as free form text entry questions We forgo a discussionof the free-form question responses due to space limitations butwe offer full anonymized participant responses in our online ap-pendix [35] We derived the Likert scale-based UX questions usingthe SUS usability scale by Brooke [16]

53 Study 3 Industrial Applicability of GvtThe goal of this final study is determine industrial applicability ofGvt To investigate this we worked with Huawei to collect twosources of information (i) the results of a survey sent to designersand developers who used Gvt in their daily developmentdesignworkflow and (ii) semi-structured interviews with both design anddevelopment managers whose teams have adopted the use of Gvt

531 Study Context ampMethodology We created a survey posingquestions related to the applicability of Gvt to industrial designersand developers These questions are shown in Fig 7 The semi-structured interviews were conducted in Chinese recorded andthen later translated During the interview managers were askedto respond to four questions related to the impact and performanceof the tool in practice We include discussions of the responses inSection 6 and stipulate full questions in our appendix

6 EMPIRICAL RESULTS61 Study 1 Results GVT PerformanceThe results of Study 1 are shown in Figure 4 This figure shows theaverageDP CP and R for each type of seeded violation over the 200seeded faults and the number of faults seeded into each category(following the distributions of our derived taxonomy) are shown onthe x-axis Overall these results are extremely encouraging withthe overall DP achieving 994 the average CP being 984 and

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden

CPDP

R

000 025 050 075 100

Figure 5 Study 2 - Developer CP DP and R

I found these reports unnecessarily complex

I found these reports very cumbersome to read

I think that I would like to use these reports frequently for Identifying Presentation Issues

I thought these reports were very useful for accurately identifying Presentation Errors

These reports are easy to readunderstand

SD D N A SA

Figure 6 Study 2 - UXQuestion Responses SD=Strongly Dis-agree D=Disagree N=Neutral A=Agree SA=Strongly Agree

the average R reaching 965 This illustrates that Gvt is capableof detecting seeded faults designed to emulate both the type anddistribution of DVs encountered in industrial settings While Gvtachieved at least 85 precision for each type of seeded DV it per-formed worse on some types of violations compared to others Forinstance Gvt saw its lowest precision values for the Font-Style andFont-Color violations typically due to the fact that the magnitudeof perturbation for the color or font type was not large enough tosurpass the Color or Image Difference Thresholds (CT amp IDT ) Gvttook 368 mins to process and generate reports for the set of 100screens with injected DVs or 22 sec per screen pair This executioncost was generally acceptable by our industrial collaborators

62 Study 2 Results GVT UtilityThe DP CP and R results representing the Android developersability to correctly detect and classify DVs is shown in Figure 5as box-plots across all 10 participants Here we found CP=DP aswhen a user misclassified violations they also did not detect themAs this figure shows the Android developers generally performedmuch worse compared to Gvt achieving an average CP of underasymp 60 and an average R of asymp 50 The sources of this performanceloss for the study participants compared to Gvt was fourfold (i)participants tended to report minor acceptable differences in fontsacross the examples (despite the instructions clearly stating notto report such violations) (ii) users tended to attribute more thanone DV to a single component specifically for font style and fontcolor violations despite instructions to report only one (iii) userstended to misclassify DVs based on the provided categories (egclassifying a layout DV for a Text GC as an incorrect text DV ) and(iv) participants missed reporting many of the injected DVs lead-ing to the low recall numbers These results indicate that at thevery least developers can struggle to both detect and classify DVsbetween mock-up and implementation screen pairs signaling theneed for an automated system to check for DVs before implementedapps are sent to a UIUX team for auditing This result confirmsthe notion that developers may not be as sensitive to small DVsin the GUI as the designers who created the GUI specificationsFurthermore this finding is notable because as part of the iterativeprocess of resolving design violations designers must communicateto developers DVs and developers must recognize and understandthese DVs in order to properly resolve them This process is oftencomplicated due to ambiguous descriptions of DVs from designers

The GVT allowed for better transfer of the design from mockminusups to the implementation of the app

The GVT has helped you to reduce the time required for verifying design violations

The GVT is able to accurately report existing design violations in productionminusquality applications

The GVT tool helped my team (designimplementation) communicate with other teams (implementationdesign)

regarding GUI design violations

Using the GUIminusVerification tool (GVT) helped to improve the quality of mobile

applications produced by industrial partner

SD D N A SA

Figure 7 Study 3 - Applicability Questions SD=Strongly Dis-agree D=Disagree N=Neutral A=Agree SA=Strongly Agree

to developers or developers disagreeing with designers over the ex-istence or type of a DV In contrast to this fragmented process Gvtprovides clear unambiguous reports that facilitate communicationbetween designers and developers

Figure 6 illustrates the responses to the likert based UX ques-tions and the results are quite encouraging In general participantsfound that the reports from Gvt were easy to read useful for iden-tifying DVs and indicated that they would like to use the reports foridentifying DVs Participants also indicated that the reports werenot unnecessarily complex or difficult to read We asked the partic-ipants about their preferences for the Gvt reports as well askingabout the most and least useful information in the reports Everysingle participant indicated that the highlighted annotations on thescreenshots in the report were the most useful element Whereasmost users tended to dislike the PID output included at the bottomof the report citing this information as difficult to comprehend

63 Study 3 Results Industrial ApplicabilityThe results for the applicability questions asked to 20 designersand developers who use Gvt in their daily activities is shown inFigure 7 A positive outcome for each of these statements correlatesto responses indicating that developers ldquoagreerdquo or ldquostrongly agreerdquoThe results of this study indicate a weak agreement of developersfor these statements indicating that while Gvt is generally appli-cable there are some drawbacks that prevented developers anddesigners from giving the tool unequivocal support We explorethese drawbacks by conducting semi-structured interviews

In conducting the interviews one of the authors asked the ques-tions presented in Figure 7 to 3 managers (2 from UIUX teams and1 from a Front-End development team) When asked whether Gvtcontributed to an increased quality of mobile applications at thecompany all three managers tended to agree that this was the caseFor instance one of the design managers stated ldquoCertainly yes Thetool is the industryrsquos first and the other designer manager addedldquoWhen the page is more complicated the tool is more helpful

When asked about the overall performance and accuracy of thetool in detecting DVs the manager from the implementation teamadmitted that the current detection performance of the tool is goodbut suggested that dynamic detection of some components mayimprove it stating ldquo[DVs ] can be detected pretty well [but the toolis] not very flexible For example a switch component in the designis open but the switch is off in the implementation He suggestedthat properly handling cases such as this would make the toolmore useful from a developers perspective One of the design teammanagers held a similar view stating that ldquoCurrently most errorsare layout errors so tool is accurate Static components are basicallydetected [but] maybe the next extension should focus on dynamic

ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk

components While the current version of the Gvt allows for theexclusion of regions with dynamic components it is clear that bothdesign and development teams would appreciate proper detectionof DVs for dynamic components Additionally two of the managerscommented on the ldquorigidityrdquo of the Gvtrsquos current interface andexplained that a more streamlined UI would help improve its utility

When asked about whether Gvt improved communication be-tween the design and development teams the development teammanager felt that while the tool has not improved communica-tion yet it did have the potential to do so ldquoAt present there is no[improvement] but certainly there is the potential possibility Thedesign managers generally stated that the tool has helped withcommunication particularly in clarifying subtle DVs that may havecaused arguments between teams in the past ldquoIf you consider thetime savings on discussion and arguments between the two teamsthis tool saves us a lot of time Another designer indicated that thetool is helpful at describing DVs to developers who may not be ableto recognize them with the naked eye ldquoWe found that the tool canindeed detect something that the naked eye cannot While there arecertainly further refinements that can be made to Gvt it is clearthat the tool has begun to have a positive impact of the developmentof mobile apps and as the tool evolves within the company shouldallow for continued improvements in quality and time saved

7 LIMITATIONS amp THREATS TO VALIDITYLimitations While we have illustrated thatGvt is applicable in anindustrial setting the tool is not without its limitations Currentlythe tool imposes lightweight restrictions on designers creatingSketch mock-ups chief among these being the requirement thatbounding boxes of components do not overlap Currently Gvt willtry to resolve such cases during the GUI-Comprehension stage usingan Intersection over union (IOU) metricInternal Validity While deriving the taxonomy of DVs mistakesin classification arising from subjectiveness may have introducedunexpected coding Tomitigate this threat we followed a setmethod-ology merged coding results and performed conflict resolutionConstruct Validity In our initial study (Sec 3) a threat to con-struct validity arises in the form of themanner in which coders wereexposed to presentation failures To mitigate this threat designersfrom our industrial partner manually annotated the screen pairsin order to clearly illustrate the affected GCs on the screen In ourevaluation of Gvt threats arise from our method of DV injectionusing the synthetic fault injection tool However we designed thistool to inject faults based upon both the type and distribution offaults from our DV taxonomy to mitigate this threatExternal Validity In our initial study related to the DV taxonomywe utilized a dataset from a single (albeit large) company withexamples across several different applications and screens There isthe potential that this may not generalize to other industrial mobileapplication development environments and platforms or mobile appdevelopment in general However given the relatively consistentdesign paradigms of mobile apps we expect the categories andthe sub-categories within the taxonomy to hold although it ispossible that the distribution across these categories may varyacross application development for different domains In Study 3we surveyed employees at a single (though large) company andfindings may differ in similar studies at other companies

8 RELATEDWORKWeb Presentation Failures The work most closely related to ourapproach are approaches that aim at detecting classifying and fixingpresentation failures in web applications [30ndash32 42] In comparisonto these approaches Gvt also performs detection and localizationof presentation failures but is the first to do so for mobile appsIn addition to the engineering challenges associated with buildingan approach to detect presentation failures in the mobile domain(eg collection and processing of GUI-related data) Gvt is the firstapproach to leverage metadata from software mock-up artifacts(eg Marketch) to perform GC matching based upon the spatialinformation collected from both mock-ups and dynamic applicationscreens allowing for precise detection of the different types of DVsdelineated in our industrial DV taxonomy Gvt is also the first toapply the processes of CQ CH analysis and B-PID toward detectingdifferences in the content and color of icons and images displayedin mobile apps Gvt also explicitly identifies and reports differentfaulty properties (such as errors in component location or text)Cross Browser Testing Approaches for XBT (or cross browsertesting) by Roy Choudhry et al [18 42 43] examine and automati-cally report differences in web pages rendered in multiple browsersThese approaches are currently not directly applicable to mock-updriven development for mobile appsVisualGUI Testing A concept known as Visual GUI Testing (VGT)aims to test certain visual aspects of a software applicationrsquos GUIas well as the underlying functional properties To accomplish thisvisual GUI testing usually executes actions on a target applicationsin order to exercise app functionality [13 14 23 38] In contrastto these approaches Gvt is designed to apply to mobile-specificDVs is tailored for the mock-up driven development practice andis aimed only at verifying visual properties of a mobile apprsquos GUIOther Approaches There are other approaches and techniquesthat related to identifying problems or differences with GUIs ofmobile apps Xie et al introduced GUIDE [47] a tool for GUI differ-encing between successive releases of GUIs for an app by matchingcomponents between GUI-hierarchies Gvt utilizes a matching pro-cedure for leaf node components as direct tree comparisons are notpossible in the context of mock-up driven development There hasalso been both commercial and academic work related to graphi-cal software built specifically for creating high-fidelity mobile appmock-ups or mockups that encode information for automated cre-ation of code for a target platform [4 8 9 34] However such toolstend to either impose too many restrictions on designers or do notallow for direct creation of code thus DVs still persist in practice9 CONCLUSION amp FUTUREWORKIn this paper we have formalized the problem of detecting designviolations in mobile apps and derived a taxonomy of design viola-tions based on a robust industrial dataset We presented Gvt anapproach for automatically detecting classifying and reporting de-sign violations in mobile apps and conducted a wide ranging studythat measured performance utility and industrial applicability ofthis tool Our results indicate that Gvt is effective in practice offersutility for developers and is applicable in industrial contexts

ACKNOWLEDGMENTSThe authors would like to thank Kebing Xie Roozbeh Farahbodand the developers and designers at Huawei for their support

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden

REFERENCES[1] Adobe photoshop httpwwwphotoshopcom[2] Android uiautomator httpdeveloperandroidcomtoolshelpuiautomator

indexhtml[3] Apple app store httpswwwapplecomiosapp-store[4] Fuild-ui httpswwwfluiduicom[5] Google play store httpsplaygooglecomstorehl=en[6] The marketch plugin for sketch httpsgithubcomtudou527marketch[7] Mobile apps What consumers really need and want httpsinfodynatracecom

rscompuwareimagesMobileAppSurveyReportpdf[8] Mockupio httpsmockupioabout[9] Protoio httpsprotoio[10] The sketch design tool httpswwwsketchappcom[11] Sketch extensions httpswwwsketchappcomextensions[12] Why your apprsquos ux is more important than you think httpwwwcodemagcom

Article1401041[13] E Aleacutegroth and R Feldt On the long-term use of visual gui testing in industrial

practice a case study Empirical Software Engineering 22(6)2937ndash2971 Dec 2017[14] E Alegroth Z Gao R Oliveira and A Memon Conceptualization and evaluation

of component-based testing unified with visual gui testing An empirical studyIn 2015 IEEE 8th International Conference on Software Testing Verification andValidation (ICST) pages 1ndash10 April 2015

[15] G Bavota M Linares-Vaacutesquez C Bernal-Caacuterdenas M Di Penta R Oliveto andD Poshyvanyk The impact of api change- and fault-proneness on the user ratingsof android apps Software Engineering IEEE Transactions on 41(4)384ndash407 April2015

[16] J Brooke SUS A quick and dirty usability scale In P W Jordan B WeerdmeesterA Thomas and I L Mclelland editors Usability evaluation in industry Taylorand Francis London 1996

[17] K Charmaz Constructing Grounded Theory SAGE Publications Inc 2006[18] S R Choudhary M R Prasad and A Orso Crosscheck Combining crawling and

differencing to better detect cross-browser incompatibilities in web applicationsIn Proceedings of the 2012 IEEE Fifth International Conference on Software TestingVerification and Validation ICST rsquo12 pages 171ndash180 Washington DC USA 2012IEEE Computer Society

[19] A Ciurumelea A SchaufelbAtildeČAcircijhl S Panichella and H C Gall Analyzingreviews and code of mobile apps for better release planning In 2017 IEEE 24th In-ternational Conference on Software Analysis Evolution and Reengineering (SANER)pages 91ndash102 Feb 2017

[20] A Di Sorbo S Panichella C V Alexandru J Shimagaki C A Visaggio G Can-fora and H C Gall What would users change in my app summarizing appreviews for recommending software changes In Proceedings of the 2016 24thACM SIGSOFT International Symposium on Foundations of Software EngineeringFSE 2016 pages 499ndash510 New York NY USA 2016 ACM

[21] K Holl and F Elberzhager A mobile-specific failure classification and its usageto focus quality assurance In 2014 40th EUROMICRO Conference on SoftwareEngineering and Advanced Applications pages 385ndash388 Aug 2014

[22] G Hu X Yuan Y Tang and J Yang Efficiently effectively detecting mobileapp bugs with appdoctor In Proceedings of the Ninth European Conference onComputer Systems EuroSys rsquo14 pages 181ndash1815 New York NY USA 2014ACM

[23] A Issa J Sillito and V Garousi Visual testing of graphical user interfaces Anexploratory study towards systematic definitions and approaches In 2012 14thIEEE International Symposium on Web Systems Evolution (WSE) pages 11ndash15 Sept2012

[24] N Jones Seven best practices for optimizing mobile testing efforts TechnicalReport G00248240 Gartner

[25] K Kuusinen and T Mikkonen Designing user experience for mobile apps Long-term product owner perspective In 2013 20th Asia-Pacific Software EngineeringConference volume 1 of APSECrsquo13 pages 535ndash540 Dec 2013

[26] V Lelli A Blouin and B Baudry Classifying and qualifying gui defects In 2015IEEE 8th International Conference on Software Testing Verification and Validation(ICST) pages 1ndash10 April 2015

[27] M Linares-Vaacutesquez G Bavota C Bernal-Caacuterdenas M Di Penta R Oliveto andD Poshyvanyk Api change and fault proneness A threat to the success ofandroid apps In Proceedings of the 2013 9th Joint Meeting on Foundations ofSoftware Engineering ESECFSErsquo13 pages 477ndash487 New York NY USA 2013ACM

[28] M Linares-Vaacutesquez G Bavota M D Penta R Oliveto and D Poshyvanyk Howdo API changes trigger Stack Overflow discussions a study on the android SDK

In Proceedings of the 22nd International Conference on Program ComprehensionICPCrsquo14 pages 83ndash94 2014

[29] M Linares-Vaacutesquez G Bavota M Tufano K Moran M Di Penta C VendomeC Bernal-Caacuterdenas and D Poshyvanyk Enabling mutation testing for androidapps In Proceedings of the 2017 11th Joint Meeting on Foundations of SoftwareEngineering ESECFSE 2017 pages 233ndash244 New York NY USA 2017 ACM

[30] S Mahajan A Alameer P McMinn and W G Halfond Automated repair oflayout cross browser issues using search-based techniques In InternationalConference on Software Testing and Analysis ISSTArsquo17 2017

[31] S Mahajan and W G J Halfond Detection and localization of html presentationfailures using computer vision-based techniques In Proceedings of the 8th IEEEInternational Conference on Software Testing Verification and Validation ICSTrsquo15April 2015

[32] S Mahajan B Li P Behnamghader and W G Halfond Using visual symptomsfor debugging presentation failures in web applications In Proceeding of the9th IEEE International Conference on Software Testing Verification and Validation(ICST) ICSTrsquo16 April 2016

[33] T McDonnell B Ray andM Kim An empirical study of api stability and adoptionin the android ecosystem In Proceedings of the 2013 International Conference onSoftware Maintenance ICSMrsquo13 pages 70ndash79 2013

[34] J Meskens K Luyten and K Coninx Plug-and-design Embracing mobiledevices as part of the design environment In Proceedings of the 1st ACM SIGCHISymposium on Engineering Interactive Computing Systems EICS rsquo09 pages 149ndash154 New York NY USA 2009 ACM

[35] K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk Gvt onlineappendix httpwwwandroid-dev-toolscomgvt

[36] P Morville User experience design httpsemanticstudioscomuser_experience_design

[37] B Myers Challenges of hci design and implementation interactions 1(1)73ndash83Jan 1994

[38] B N Nguyen B Robbins I Banerjee and A Memon Guitar An innovativetool for automated testing of gui-driven software Automated Software Engg21(1)65ndash105 Mar 2014

[39] T A Nguyen and C Csallner Reverse engineering mobile application userinterfaces with REMAUI In Proceedings of the 2015 30th IEEEACM Interna-tional Conference on Automated Software Engineering ASErsquo15 pages 248ndash259Washington DC USA 2015 IEEE Computer Society

[40] F PalombaM Linares-Vaacutesquez G Bavota R Oliveto M D Penta D Poshyvanykand A D Lucia User reviews matter tracking crowdsourced reviews to supportevolution of successful apps In 2015 IEEE International Conference on SoftwareMaintenance and Evolution (ICSME) pages 291ndash300 Sept 2015

[41] F Palomba P Salza A Ciurumelea S Panichella H Gall F Ferrucci andA De Lucia Recommending and localizing change requests for mobile appsbased on user reviews In Proceedings of the 39th International Conference onSoftware Engineering ICSE rsquo17 pages 106ndash117 Piscataway NJ USA 2017 IEEEPress

[42] S Roy Choudhary M R Prasad and A Orso X-pert Accurate identification ofcross-browser issues in web applications In Proceedings of the 2013 InternationalConference on Software Engineering ICSE rsquo13 pages 702ndash711 Piscataway NJUSA 2013 IEEE Press

[43] S Roy Choudhary H Versee and A Orso Webdiff Automated identification ofcross-browser issues in web applications In Proceedings of the 2010 IEEE Interna-tional Conference on Software Maintenance ICSM rsquo10 pages 1ndash10 WashingtonDC USA 2010 IEEE Computer Society

[44] T Silva da Silva A Martin F Maurer and M Silveira User-centered design andagile methods A systematic review In Proceedings of the 2011 Agile ConferenceAGILE rsquo11 pages 77ndash86 Washington DC USA 2011 IEEE Computer Society

[45] K-J Stol P Ralph and B Fitzgerald Grounded theory in software engineeringresearch A critical review and guidelines In Proceedings of the 38th InternationalConference on Software Engineering ICSE rsquo16 pages 120ndash131 New York NY USA2016 ACM

[46] A B Tucker Computer Science Handbook Second Edition Chapman amp HallCRC2004

[47] Q Xie M Grechanik C Fu and C Cumby Guide A gui differentiator In 2009IEEE International Conference on Software Maintenance ICSMrsquo09

[48] H Yee S Pattanaik and D P Greenberg Spatiotemporal sensitivity and visualattention for efficient rendering of dynamic environments ACM Trans Graph20(1)39ndash65 Jan 2001

[49] C Zeidler C Lutteroth W Stuerzlinger and G Weber Evaluating Direct Manip-ulation Operations for Constraint-Based Layout pages 513ndash529 Springer BerlinHeidelberg Berlin Heidelberg 2013

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden

A) App Mock-Up B) App Implementation

GUI-Component

Screen

Presentation Failurewith three DesignViolations (Color Size Location)

Presentation Failurewith one DesignViolation (Font)

Figure 1 Examples of Formal Definitions

Definition 2 Screen (S) - A canvas S with a predefined heightand width corresponding to the physical display dimensions of asmartphone or tablet Each Screen contains a cumulative hierarchyof components which can be represented as a nested set such that

S = GC1GC2GCi GC3 (1)

where each GC has a unique attribute tuple and the nested set canbe ordered in either depth-first (Exp 1) or in a breadth-first mannerWe are concerned with two specific types of screens screens repre-senting mock-ups of mobile apps Sm and screens representing realimplementations of these apps or Sr

212 Design Violations amp Presentation Failures As describedearlier design violations correspond to visual symptoms of pre-sentation failures or differences between the intended design andimplementation of a mobile app screen Presentation failures canbe made up of one or more design violations of different typesDefinition 3 Design Violation (DV) - As shown in Exp 2 amismatch between the attribute tuples of two corresponding leaf-level (ie having no direct children) GUI-componentsGCmi andGCrjof two screens Sm and Sr imply a design violation DV associatedwith those components In this definition leaf nodes correspond toone another if their location and size on a screen (ie ltx-positiony-positiongt ltheightwidthgt) match within a given threshold Equalitybetween leaf nodes is measured as a tighter matching thresholdacross all attributes As we illustrate in the next section inequalitiesbetween different attributes in the associated tuples of the GCs leadto different types of design violations

(GCmi asymp GCrj ) and (GCmi GCrj )

=rArr DV isin GCmi GCrj

(2)

Definition 4 Presentation Failure (PF) - A set of one or moreDVs attributed to a set of corresponding GUI-components betweentwo screens Sm and Sr as shown in Exp 3 For instance as shownin Fig 1 a single set of corresponding components may have dif-ferences in both the ltxygt and ltheightwidthgt attributes leading totwo constituent design violations that induce a single presentationfailure PF Thus each presentation failure between two Screens Scorresponds to at least one mismatch between the attribute vectorsof two corresponding leaf node GUI-components GCim and GCir

if DV1DV2 DVi isin GCmi GCrj

then PF isin Sm Sr (3)

213 Problem Statement Given these definitions the problembeing solved in this paper is the following Given two screens Smand Sr corresponding to the mock-up and implementation screensof a mobile application we aim to detect and describe the set ofpresentation failures PF1 PF2 PFi isin Sm Sr Thus we aimto report all design violations on corresponding GC pairs

DV1DV2 DVk isinGCmi1 GC

rj1 GC

mi2 GC

rj2 GC

mix GC

rjy

(4)

22 Industrial Problem OriginsA typical industrial mobile development process includes the fol-lowing steps (as confirmed by our collaborators at Huawei) (i)First a team of designers creates highly detailed mockups of anapprsquos screens using the Sketch [10] (or similar) prototyping soft-ware These mock-ups are typically ldquopixel-perfect representationsof the app for a given screen dimension (ii) The mock-ups arethen handed off to developers in the form of exported images withdesigner added annotations stipulating spatial information andconstraints Developers use this information to implement repre-sentations of the GUIs for Android using a combination of Javaand xml (iii) Next after the initial version of the app has beenimplemented compiled Android Application Package(s) (ie apks)are sent back to the designers who then install these apps on targetdevices generate screenshots for the screens in question and man-ually search for discrepancies compared to the original mock-ups(iv) Once the set of violations are identified these are communi-cated back to the developers via textual descriptions and annotatedscreenshots at the cost of significant manual effort from the designteams Developers must then identify and resolve the DVs usingthis information The process is often repeated in several iterationscausing substantial delays in the development process

The goal of our work is to drastically improve this iterativeprocess by (i) automating the identification of DVs on the screensof mobile apps - saving both the design and development teamstime and effort and (ii) providing highly accurate information tothe developers regarding these DVs in the form of detailed reports -in order to reduce their effort in resolving the problem

3 DESIGN VIOLATIONS IN PRACTICEIn order to gain a better understanding of the types ofDVs that occurin mobile apps in practice we conducted a study using a datasetfrom Huawei While there do exist a small collection of taxonomiesrelated to visual GUI defects [23 26] and faults in mobile apps [2129] we chose to conduct a contextualized study with our industrialpartner for the following reasons (i) existing taxonomies for visualGUI defects were not detailed enough containing only generalfaults (eg ldquoincorrect appearancerdquo) (ii) existing fault taxonomiesfor mobile apps either did not contain visual GUI faults or werenot complete and (iii) we wanted to derive a contextualized DVtaxonomy for apps developed at Huawei The findings from thisstudy underscore the existence and importance of the problem thatour approach aims to solve in this context Due to an NDA we arenot able to share the dataset or highlight specific examples in orderto avoid revealing information about future products at HuaweiHowever we present aggregate results in this section

ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk

Layout Violations

39

Text Violations

23

Resource Violations

38

Horizontal Translation

10

Vertical Translation

16HampVSize5

H Size4

HampVTrans

5

Font Style Violation

8

Font Color Violation

9

Incorrect Text Violation

6

Image Color Violation

10

Missing Component12

Extra Component

6

Incorrect Image

7

Component Shape 1 Component Type 1

H = Horizontal V = VerticalTrans = Translation

Figure 2 Distribution of Different Types of Industrial DVs

31 Study Setting amp MethodologyThe goal of this study is to derive a taxonomy of the differenttypes of DVs and examine the distribution of these types inducedduring the mobile app development process The context of thisstudy is comprised of a set of 71 representative mobile app mock-up and implementation screen pairs from more than 12 differentinternal apps annotated by design teams from our industrial partnerto highlight specific instances of resolved DVs This set of screenpairs was specifically selected by the industrial design team tobe representative both in terms of diversity and distribution ofviolations that typically occur during the development process

In order to develop a taxonomy and distribution of the violationspresent in this dataset we implement an open coding methodologyconsistent with constructivist grounded theory [17] Following theadvice of recent work within the SE community [45] we stipulateour specific implementation of this type of grounded theory whilediscussing our deviations from the methods in the literature Wederived our implementation from the material discussed in [17]involving the following steps (i) establishing a research problemand questions (ii) data-collection and initial coding and (iii) focusedcoding We excluded other steps described in [17] such as memoingbecause we were building a taxonomy of labels and seeking newspecific data due to our NDA limiting the data that could be sharedThe study addressed the following research question What are thedifferent types and distributions of GUI design violations that occurduring industrial mobile app development processes

During the initial coding process three of the authors were sentthe full set of 71 screen pairs and were asked to code four pieces ofinformation for each example (i) a general category for the viola-tion (ii) a specific description of the violation (iii) the severity ofthe violation (if applicable) and (iv) the Android GC types affected(eg button) Finally we performed a second round of coding thatcombined the concepts of focused and axial coding as described in[17] During this round two of the authors merged the responsesfrom all three types of coding information where at least two ofthe three coders agreed During this phase similar coding labels

were merged (eg ldquolayout violation vs ldquospatial violation) conflictswere resolved two screen pairs were discarded due to ambiguityand cohesive categories and subcategories were formed The authoragreement for each of the four types of tags is as follows (i) generalviolation category (100) (ii) specific violation description (96)(iii) violation severity (100) and (iv) affected GC types (845)32 Grounded Theory Study ResultsOur study revealed three major categories of design violationseach with several specific subtypes We forgo detailed descriptionsand examples of violations due to space limitations but provideexamples in our online appendix [35] The derived categories andsubcategories of DVs and their distributions are illustrated in Fig 2Overall 82 DVs were identified across the 71 unique screen pairsconsidered in our study The most prevalent category of DVs in ourtaxonomy are Layout Violations (asymp 40) which concern either atranslation of a component in the x or y direction or a change in thecomponent size with translations being more common The secondmost prevalent category (asymp 36) consists of Resource Violationswhich concern missing components extra components color dif-ferences and image differences Finally about one-quarter (asymp 24)of these violations are Text Violations which concern differences incomponents that display text We observed that violations typicallyonly surfaced for ldquoleaf-level components in the GUI hierarchyThat is violations typically only affected atomic components ampnot containers or backgrounds Only 582 of examined violations(asymp 6) affected backgrounds or containers Even in these few casesthe violations also affected ldquoleaf-level components

The different types of violations correspond to different inequali-ties between the attribute tuples of corresponding GUI-componentsdefined in Sec 2 This taxonomy shows that designers are chargedwith identifying several different types of design violations a daunt-ing task particularly for hundreds of screens across several apps

4 THE GVT APPROACH41 Approach OverviewThe workflow of Gvt (Fig 3) proceeds in three stages First in theGUI-Collection Stage GUI-related information from both mock-upsand running apps is collected Next in theGUI-Comprehension Stageleaf-level GCs are parsed from the trees and a KNN-based algorithmis used to match corresponding GCs using spatial information Fi-nally in the Design Violation Detection Stage DVs are detected usinga combination of methods that leverage spatial GC information andcomputer vision techniques42 Stage 1 GUI Collection

421 Mock-Up GUI Collection Software UIUX design profes-sionals typically use professional-grade image editing software(such as Photoshop[1] or Sketch[10]) to create their mock-ups De-signers employed by our industrial partner utilize the Sketch designsoftware Sketch is popular among mobile UIUX designers due toits simple but powerful features ease of use and large library ofextensions [11] When using these tools designers often constructgraphical representations of smartphone applications by placingobjects representing GCs (which we refer to as mock-up GCs) on acanvas (representing a Screen S) that matches the typical displaysize of a target device In order to capture information encodedin these mock-ups we decided to leverage an export format that

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden3 Design Violation Detection

2 GUI-Comprehension11 Mock-Up GUI Collection

12 App Implementation GUI Collection

Resource Violation Detector

Text Violation Detector

GVT GUI Collector

apk

orapp src

Physical Deviceor Emulator

Android UIAutomator

Android adb

Screencap

AndroidApplication

Marketch Javascript

File+

+

Export Screenshot

and Marketch

Files

Mock-Up GUI Hierarchy

Implementation GUI Hierarchy

UIX Parser

GUI Hierarchy

Construction

Detection of Corresponding

Leaf Node Pairs

Identified Corresponding Component Pairs

Difference Image

Layout Violation Detector

Sketch GUI-Parser

Perceptual Image

Differencing (PID)

UIXxml File

Checksltimagegtlttextgt

Color Quantization amp Histogram Analysis

Checksltimagegt

+Missing amp Extraneous

Components

Color Quantization amp Histogram Analysis

Javascript Parser

GUIHierarchy

Construction

UIX Parser

Violation Manager

Checksltwidthgtltheighgt

Checksltxgtltygt

Design Violation Resolver

Difference Image Parser

MismatchedComponentPairs

ne

Spatial Comparator

Size Comparator

Component-LevelPerceptual Image Diff

Figure 3 Overview of GVTWorkflowwas already in use by our industrial partner an open-source Sketchextension called Marketch [6] that exports mock-ups as an html

page including a screenshot and JavaScript fileThus as input from the mock-up Gvt receives a screenshot (to

be used later in the Design Violation Detection Phase) and a directorycontaining the Marketch information The JavaScript file containsseveral pieces of information for each mock-up GC including (i) thelocation of the mock-up GC on the canvas (ii) size of the boundingbox and (iii) the textfont displayed by the mock-up GC (if any) Asshown in Figure 3- 11 we built a parser to read this informationHowever it should be noted that our approach is not tightly coupled toSketch or Marketch files1 After the Marketch files have been parsedGvt examines the extracted spatial information to build a GC hier-archy The result can be logically represented as a rooted tree whereleaf nodes contain the atomic UI-elements with which a typicaluser might interact Non-leaf node components typically representcontainers that form logical groupings of leaf node componentsand other containers In certain cases our approximation of usingmock-up GCs to represent implementation GCs may not hold Forinstance an icon which should be represented as a single GC mayconsist of several mock-up GCs representing parts of the icon Gvthandles such cases in the GUI-Comprehension stage

422 Dynamic App GUI-Collection In order to compare thethe mock-up of an app to its implementation Gvt must extractGUI-related meta-data from a running Android app Gvt is able touse Androidrsquos uiautomator framework [2] intended for UI testingto capture xml files and screenshots for a target screen of an apprunning on a physical device or emulator Each uiautomator filecontains information related to the runtime GUI-hierarchy of thetarget app including the following attributes utilized by Gvt (i)The Android component type (eg androidwidgetImageButton)(ii) the location on the screen (iii) the size of the bounding box (iv)text displayed (v) a developer assigned id The hierarchal structureof components is encoded directly in the uiautomator file and thuswe built a parser to extract GUI-hierarchy using this informationdirectly (see Fig 3- 12 )

43 Stage 2 GUI ComprehensionIn order for Gvt to find visual discrepancies between componentsexisting in the mock-up and implementation of an app it must1Similar information regarding mock-up GCs can be parsed from the html or ScalableVector Graphics (svg) format exported by other tools such as Photoshop[1]

determine which components correspond to one another Unfor-tunately the GUI-hierarchies parsed from both the Marketch anduiautomator files tend to differ dramatically due to several factorsmaking tree-based GC matching difficult First since the hierarchyconstructed using the Marketch files is generated using informationfrom the Sketch mock-up of app it is using information derivedfrom designers While designers have tremendous expertise in con-structing visual representations of apps they typically do not takethe time to construct programmatically-oriented groupings of com-ponents Furthermore designers are typically not aware of thecorrect Android component types that should be attributed to dif-ferent objects in a mock-up Second the uiautomator representationof the GUI-hierarchy contains the runtime hierarchal structure ofGCs and correct GC types This tree is typically far more com-plex containing several levels of containers grouping GCs togetherwhich is required for the responsive layouts typical of mobile apps

To overcome this challenge Gvt instead forms two collectionsof leaf-node components from both the mock-up and implementa-tion GUI-hierarchies (Fig 3- 2 ) as this information can be easilyextracted As we reported in Sec 3 the vast majority of DVs af-fects leaf-node components Once the leaf node components havebeen extracted from each hierarchy GVT employs a K-Nearest-Neighbors (KNN) algorithm utilizing a similarity function basedon the location and size of the GCs in order to perform matchingIn this setting an input leaf-node component from the mock-upwould be matched against it closest (eg K=1) neighbor from theimplementation based upon the following similarity function

γ = (|xm minus xr | + |ym minus yr | + |wm minuswr | + |hm minus hr |) (5)

Where γ is a similarity score where smaller values represent closermatches The x yw and h variables correspond to the x amp y lo-cation of the top and left-hand borders of the bounding box andthe height and width of the bounding boxes for the mock-up andimplementation GCs respectively The result is a list of GCs thatshould logically correspond to one another (corresponding GCs)

It is possible that there exist instances of missing or extraneouscomponents between the mock-up and implementation To identifythese cases our KNN algorithm employs a GC-Matching Threshold(MT ) If the similarity score of the nearest neighbor match for agiven input mock-up GC exceeds this threshold it is not matchedwith any component and will be reported as amissing GC violation

ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk

If there are unmatched GCs from the implementation they are laterreported as extraneous GC violations

Also there may be cases where a logical GC in the implemen-tation is represented as small group of mock-up GCs Gvt is ableto handle these cases using the similarity function outlined aboveFor each mock-up GC Gvt checks whether the neighboring GCsin the mockup are closer than the closest corresponding GC inthe implementation If this is the case they are merged with theprocess repeating until a logical GUI-component is represented44 Stage 3 Design Violation DetectionIn the Design Violation Detection stage of the Gvt workflow theapproach uses a combination of computer vision techniques andheuristic checking in order to effectively detect the different cate-gories of DVs derived in our taxonomy presented in Section 3

441 Perceptual Image Differencing In order to determine cor-responding GCs with visual discrepancies Gvt uses a techniquecalled Perceptual Image Differencing (PID) [48] that operates uponthe mock-up and implementation screenshots PID utilizes a modelof the human visual system to compare two images and detectvisual differences and has been used to successfully identify visualdiscrepancies in web applications in previous work [31 32] Weuse this algorithm in conjunction with the GC information derivedin the previous steps of Gvt to achieve accurate violation detec-tion For a full description of the algorithm we refer readers to[48] The PID algorithm uses several adjustable parameters includ-ing F which corresponds to the visual field of view in degrees Lwhich indicates the luminance or brightness of the image and Cwhich adjusts sensitivity to color differences The values used inour implementation are stipulated in Section 45

The output of the PID algorithm is a single difference image (Fig3- 3 ) containing difference pixels which are pixels considered to beperceptually different between the two images After processing thedifference image generated by PID Gvt extracts the implementa-tion bounding box for each corresponding pair of GCs and overlaysthe box on top of the generated difference image It then calculatesthe number of difference pixels contained within the bounding boxwhere higher numbers of difference pixels indicate potential visualdiscrepancies Thus Gvt collects all ldquosuspicious GC pairs with a of difference pixels higher than a Difference Threshold DT This setof suspicious components is then passed to the Violation Manager(Fig 3- 3 ) so that specific instances of DVs can be detected

442 Detecting Layout Violations The first general category ofDVs thatGvt detects are Layout Violations According the taxonomyderived in Sec 3 there are six specific layout DV categories thatrelate to two component properties (i) screen location (ie ltxygtposition) and (ii) size (ie lthwgt of the GC bounding box) Gvt firstchecks for the three types of translationDVs utilizing a heuristic thatmeasures the distance from the top and left-hand edges of matchedcomponents If the difference between the components in either thex ory dimension is greater than a Layout Threshold (LT ) then thesecomponents are reported as a Layout DV Using the LT avoids triviallocation discrepancies within design tolerances being reported asviolations and can be set by a designer or developer using the toolWhen detecting the three types of size DVs in the derived designviolation taxonomyGvt utilizes a heuristic that compares thewidthand height of the bounding boxes of corresponding components If

the width or height of the bounding boxes differ by more than theLT then a layout violation is reported

443 Detecting Text Violations The next general type of DVthat Gvt detects are Text Violations of which there are three spe-cific types (i) Font Color (ii) Font Style and (iii) Incorrect TextContent These detection strategies are only applied to pairs oftext-based components as determined by uiautomator informationTo detect font color violations Gvt extracts cropped images foreach pair of suspicious text components by cropping the mock-upand implementation screenshots according to the componentrsquos re-spective bounding boxes Next Color Quantization (CQ) is appliedto accumulate instances of all unique RGB values expressed in thecomponent-specific images This quantization information is thenused to construct a Color Histogram (CH) (Fig 3- 3 ) Gvt computesthe normalized Euclidean distance between the extracted ColorHistograms for the corresponding GC pairs and if the Histogramsdo not match within a Color Threshold (CT) then a Font-Color DVis reported and the top-3 colors (ie centroids) from each CH arerecorded in theGvt report Likewise if the colors domatch then thePID discrepancy identified earlier is due to the Font-Style changing(provided no existing layout DVs) and thus a Font-Style Violationis reported Finally to detect incorrect text content Gvt utilizesthe textual information preprocessed to remove whitespace andnormalize letter cases and performs a string comparison If thestrings do not match then an Incorrect Text Content DV is reported

444 Detecting Resource Violations Gvt is able to detect thefollowing resource DVs (i) missing component (ii) extraneous com-ponent (iii) image color (iv) incorrect images and (v) componentshape The detection and distinction between Incorrect Image DVsand Image Color DVs requires an analysis that combines two differ-ent computer vision techniques To perform this analysis croppedimages from the mock-up and implementation screenshots accord-ing to corresponding GCs respective bounding boxes are extractedThe goal of this analysis is to determine when the content of image-based GCs differ as opposed to only the colors of the GCs differingTo accomplish this Gvt leverages PID applied to extracted GCimages converted to a binary color space (B-PID) in order to detectdifferences in content and CQ and CH analysis to determine differ-ences in color (Sec 443) To perform the B-PID procedure croppedGC images are converted to a binary color space by extracting pixelintensities and then applying a binary transformation to the inten-sity values (eg converting the images to intensity independentblack amp white) Then PID is run on the color-neutral version ofthese images If the images differ by more than an Image DifferenceThreshold (IDT ) then an Incorrect Image DV (which encompassesthe Component Shape DV ) is reported If the component passes thebinary PID check then Gvt utilizes the same CQ and CH process-ing technique described above to detect image color DVs Missingand extraneous components are detected as described in Sec 43

445 Generating Violation Reports In order to provide develop-ers and designers with effective information regarding the detectedDVs Gvt generates an html report that for each detected viola-tion contains the following (i) a natural language description ofthe design violation(s) (ii) an annotated screenshot of the app im-plementation with the affected GUI-component highlighted (iii)cropped screenshots of the affected GCs from both the design and

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden

implementation screenshots (iv) links to affected lines of applica-tion source code (v) color information extracted from the CH forGCs identified to have color mismatches and (vi) the differenceimage generated by PID The source code links are generated bymatching the ids extracted from the uiautomator information backto their declarations in the layout xml files in the source code (egthose located in the res directory of an apprsquos source code) Weprovide examples of generated reports in our online appendix [35]45 Implementation amp Industrial CollaborationOur implementation of Gvt was developed in Java with a SwingGUI In addition to running the Gvt analysis the tool executableallows for one-click capture of uiautomator files and screenshotsfrom a connected device or emulator Several acceptance tests ofmock-upimplementation screen pairs with pre-existing violationsfrom apps under development within our industrial partner wereused to guide the development of the tool 12 Periodic releases ofbinaries for bothWindows and Mac were made to deploy the tool todesigners and developers within the company The authors of thispaper held regular bi-weekly meetings with members of the designand development teams to plan features and collect feedback

Using the acceptance tests and feedback from our collaboratorswe tuned the various thresholds and parameters of the tool for bestperformance The PID algorithm settings were tuned for sensitivityto capture subtle visual inconsistencies which are then later filteredthrough additional CV techniques F was set to 45 L was set to100cdm2 and C was set to 1 The GC-Matching Threshold (MC)was set to 18th the screen width of a target device the DT fordetermining suspicious GCs was set to 20 The LT was set to 5pixels (based on designer preference) the CT which determinesthe degree to which colors must match for color-based DVs was setto 85 and finally the IDT was set to 20 Gvt allows for a userto change these settings if desired additionally users are capableof defining areas of dynamic content (eg loaded from networkactivity) which should be ignored by the Gvt analysis5 DESIGN OF THE EXPERIMENTSTo evaluate Gvtrsquos performance usefulness and applicability weperform three complimentary studies answering the following RQs

bull RQ1 How well does Gvt perform in terms of detecting andclassifying design violations

bull RQ2 What utility can Gvt provide from the viewpoint ofAndroid developers

bull RQ3What is the industrial applicability of Gvt in terms ofimproving the mobile application development workflow

RQ1 and RQ2 focus on quantitatively measuring the performanceof Gvt and the utility it provides to developers through a controlledempirical and a user study respectively RQ3 reports the results ofa survey and semi-structured interviews with our collaboratorsaimed at investigating the industrial applicability of Gvt51 Study 1 Gvt Effectiveness amp PerformanceThe goal of the first study is to quantitatively measure Gvt in termsof its precision and recall in both detecting and classifying DVs

511 Study Context To carry out a controlled quantitativestudy we manually reverse engineered Sketch mockups for tenscreens for eight of the most popular apps on Google Play To de-rive this set we downloaded the top-10 apps from each category on

the Google-Play store removing the various categories correspond-ing to games (as these have non-standard GUI-components thatGvt does not support) We then randomly sampled one app fromeach of the remaining 33 categories eliminating duplicates (sinceapps can belong to more than one category) We then manuallycollected screenshots and uiautomator files from two screens foreach application using a Nexus 5 attempting to capture the ldquomainrdquoscreen that a user would typically interact with and one secondaryscreen Using the uiautomator files we generated cropped screen-shots of all the leaf nodes components for each screen of the appFrom these we were able generate 10 screens from 8 applicationsthat successfully ran through Gvt without any reported violations

512 Synthetic DV Injection With a set of correct mock-ups cor-responding to implementation screens in an app we needed a suit-able method to introduce DVs into our subjects To this end we con-structed a synthetic DV injection tool that modifies the uiautomator

xml files and corresponding screenshots in order to introduce de-sign violations from our taxonomy presented in Sec 3 The toolis composed of two components (i) an XML Parser that reads andextracts components from the screen then (ii) a Violation Generatorthat randomly selects components and injects synthetic violationsWe implemented injection for the following types of DVsLocation Violation The component is moved either horizontallyvertically or in both directions within the same container Howeverthemaximum distance from the original point is limited by a quarterof the width of the screen size This was based on the severity ofLayout Violations in our study described in Section 3 In order togenerate the image we cropped the component and moved it to thenew location replacing all the original pixels by the most prominentcolor from the surroundings in the original locationSize Violation The component size either increases or decreasesby 20 of the original size For instances where the component sizedecreases we replaced all the pixels by the most prominent colorfrom the surroundings of the original sizeMissing Component Violation This violation removes a leafcomponent from the screen replacing the original pixels by themost prominent color from the surrounding backgroundImage Violation We perturb 40 of the pixels in an image byrandomly generating an RGB value for the pixels affectedImage Color Violation This rule perturbs the color of an imageby shifting the hue of image colors by 30degComponent Color Violation This uses the same process as forImage Color Violations but we change the color by 180degFont Violation This violation randomly selects a font from theset of Arial Comic Sans MS Courier Roboto or Times Roman andapplies it to a TextView componentFont Color Violation changes the text color of a TextView com-ponent We extracted the text color using CH analysis then wechanged the color using same strategy as for Image Color Violations

513 Study Methodology In injecting the synthetic faults wetook several measures to simulate the creation of realistic faultsFirst we delineated 200 different types of design violations accord-ing to the distribution defined in our DV taxonomy in Sec 3 Wethen created a pool of 100 screens by creating random copies ofthe both the uiautomator xml files and screenshots from our initialset of 10 screens We then used the synthetic DV injection tool toseed faults into the pool of 100 screens according to the following

ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk

criteria (i) No screen can contain more than 3 injected DVs (ii)each GC should have a maximum of 1 DV injected and (iii) Eachscreen must have at least 1 injected DV After the DVs were seededeach of the 100 screens and 200 DVs were manually inspected forcorrectness Due to the random nature of the tool a small num-ber of erroneous DVs were excluded and regenerated during thisprocess (eg color perturbed to perceptually similar color) Thebreakdown of injected DVs is shown in Figure 4 and the full datasetwith description is included in our online appendix [35]

Once the final set of screens with injected violations was derivedwe ran Gvt across these subjects and measured four metrics (i)detection precision (DP ) (ii) classification precision (CP ) (iii) recall(R) and (iv) execution time per screen (ET ) We make a distinctionbetween detection and classification in our dataset because it is pos-sible thatGvt is capable of detecting but misclassifying a particularDV (eg an image color DV misclassified as an incorrect image DV )DP CP and R were measured according to the following formulas

DP CP =Tp

Tp + FpR =

Tp

Tp + Fn(6)

where for DP Tp represent injected design violations that weredetected and for CP Tp represents injected violations that wereboth detected and classified correctly In each case Fp correspondto detected DVs that were either not injected or misclassified ForRecallTp represents injected violations that were correctly detectedand Fn represents injected violations that were not detected Tocollect these measures two authors manually examined the reportsfrom Gvt in order to collect the metrics

52 Study 2 Gvt UtilitySince the ultimate goal of an approach like Gvt is to improvethe workflow of developers the goal of this second study is tomeasure the utility (ie benefit) that Gvt provides to developersby investigating two phenomena (i) The accuracy and effort ofdevelopers in detecting and classifying DVs and (ii) the perceivedutility of Gvt reports in helping to identify and resolve DVs

521 Study Context We randomly derived two sets of screensto investigate the two phenomena outlined above First we ran-domly sampled two mutually exclusive sets of 25 and 20 screensrespectively from the 100 used in Study 1 ensuring at least oneinstance of each type of DV was included in the set This resulted inboth sets of screens containing 40 design violations in total The cor-rect mockup screenshot corresponding to each screen sampled fromthe study were also extracted creating pairs of ldquocorrect mockupand ldquoincorrect implementation screenshots 10 participants withat least 5 years of Android development experience were contactedvia email to participate in the survey

522 Study Methodology We created an online survey withfour sections In the first section participants were given back-ground information regarding the definition of DVs and the differ-ent types of DVs derived in our taxonomy In the second sectionparticipants were asked about demographic information such asprogramming experience and education level In the third sectioneach participant was exposed to 5 mock-up implementation screenpairs (displayed side by side on the survey web page) and asked toidentify any observed design violations Descriptions of the DVswere given at the top of this page for reference For each screenpair participants were presented with a dropdown menu to select

80

85

90

95

100

Font

Colo

r(18)

Font

Styl

e(18

)

Imag

e Colo

r(22)

Imag

e(16

)

Miss

ing G

C(26)

Size(1

8)

Text

Conte

nt(1

4)

Trans

lation

(68)

DV Type

Per

cent

age

MetricCPDPR

Figure 4 Study 1 - Detection Precision (DP ) ClassificationPrecision (CP ) and Recall (R)a type for an observed DV and a text field to describe the errorin more detail For each participant one of the 5 mock-up screenswas a control containing no injected violations The 25 screenswere assigned to participants such that each screen was observedby two participants and the order of the screens presented to eachparticipant was randomized to avoid bias To measure the effective-ness of participants in detecting and describing DVs we leveragethe DP CP and R metrics introduced in Study 1 In the fourth sec-tion participants were presented with two screen pairs from thesecond set of 20 sampled from the user study as well as the Gvtreports for these screens Participants were then asked to answer5 user-preferences (UP) and 5 user experience (UX) questions aboutthese reports which are presented in the following section The UPquestions were developed according to the user experience hon-eycomb originally developed by Morville [36] and were posed toparticipants as free form text entry questions We forgo a discussionof the free-form question responses due to space limitations butwe offer full anonymized participant responses in our online ap-pendix [35] We derived the Likert scale-based UX questions usingthe SUS usability scale by Brooke [16]

53 Study 3 Industrial Applicability of GvtThe goal of this final study is determine industrial applicability ofGvt To investigate this we worked with Huawei to collect twosources of information (i) the results of a survey sent to designersand developers who used Gvt in their daily developmentdesignworkflow and (ii) semi-structured interviews with both design anddevelopment managers whose teams have adopted the use of Gvt

531 Study Context ampMethodology We created a survey posingquestions related to the applicability of Gvt to industrial designersand developers These questions are shown in Fig 7 The semi-structured interviews were conducted in Chinese recorded andthen later translated During the interview managers were askedto respond to four questions related to the impact and performanceof the tool in practice We include discussions of the responses inSection 6 and stipulate full questions in our appendix

6 EMPIRICAL RESULTS61 Study 1 Results GVT PerformanceThe results of Study 1 are shown in Figure 4 This figure shows theaverageDP CP and R for each type of seeded violation over the 200seeded faults and the number of faults seeded into each category(following the distributions of our derived taxonomy) are shown onthe x-axis Overall these results are extremely encouraging withthe overall DP achieving 994 the average CP being 984 and

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden

CPDP

R

000 025 050 075 100

Figure 5 Study 2 - Developer CP DP and R

I found these reports unnecessarily complex

I found these reports very cumbersome to read

I think that I would like to use these reports frequently for Identifying Presentation Issues

I thought these reports were very useful for accurately identifying Presentation Errors

These reports are easy to readunderstand

SD D N A SA

Figure 6 Study 2 - UXQuestion Responses SD=Strongly Dis-agree D=Disagree N=Neutral A=Agree SA=Strongly Agree

the average R reaching 965 This illustrates that Gvt is capableof detecting seeded faults designed to emulate both the type anddistribution of DVs encountered in industrial settings While Gvtachieved at least 85 precision for each type of seeded DV it per-formed worse on some types of violations compared to others Forinstance Gvt saw its lowest precision values for the Font-Style andFont-Color violations typically due to the fact that the magnitudeof perturbation for the color or font type was not large enough tosurpass the Color or Image Difference Thresholds (CT amp IDT ) Gvttook 368 mins to process and generate reports for the set of 100screens with injected DVs or 22 sec per screen pair This executioncost was generally acceptable by our industrial collaborators

62 Study 2 Results GVT UtilityThe DP CP and R results representing the Android developersability to correctly detect and classify DVs is shown in Figure 5as box-plots across all 10 participants Here we found CP=DP aswhen a user misclassified violations they also did not detect themAs this figure shows the Android developers generally performedmuch worse compared to Gvt achieving an average CP of underasymp 60 and an average R of asymp 50 The sources of this performanceloss for the study participants compared to Gvt was fourfold (i)participants tended to report minor acceptable differences in fontsacross the examples (despite the instructions clearly stating notto report such violations) (ii) users tended to attribute more thanone DV to a single component specifically for font style and fontcolor violations despite instructions to report only one (iii) userstended to misclassify DVs based on the provided categories (egclassifying a layout DV for a Text GC as an incorrect text DV ) and(iv) participants missed reporting many of the injected DVs lead-ing to the low recall numbers These results indicate that at thevery least developers can struggle to both detect and classify DVsbetween mock-up and implementation screen pairs signaling theneed for an automated system to check for DVs before implementedapps are sent to a UIUX team for auditing This result confirmsthe notion that developers may not be as sensitive to small DVsin the GUI as the designers who created the GUI specificationsFurthermore this finding is notable because as part of the iterativeprocess of resolving design violations designers must communicateto developers DVs and developers must recognize and understandthese DVs in order to properly resolve them This process is oftencomplicated due to ambiguous descriptions of DVs from designers

The GVT allowed for better transfer of the design from mockminusups to the implementation of the app

The GVT has helped you to reduce the time required for verifying design violations

The GVT is able to accurately report existing design violations in productionminusquality applications

The GVT tool helped my team (designimplementation) communicate with other teams (implementationdesign)

regarding GUI design violations

Using the GUIminusVerification tool (GVT) helped to improve the quality of mobile

applications produced by industrial partner

SD D N A SA

Figure 7 Study 3 - Applicability Questions SD=Strongly Dis-agree D=Disagree N=Neutral A=Agree SA=Strongly Agree

to developers or developers disagreeing with designers over the ex-istence or type of a DV In contrast to this fragmented process Gvtprovides clear unambiguous reports that facilitate communicationbetween designers and developers

Figure 6 illustrates the responses to the likert based UX ques-tions and the results are quite encouraging In general participantsfound that the reports from Gvt were easy to read useful for iden-tifying DVs and indicated that they would like to use the reports foridentifying DVs Participants also indicated that the reports werenot unnecessarily complex or difficult to read We asked the partic-ipants about their preferences for the Gvt reports as well askingabout the most and least useful information in the reports Everysingle participant indicated that the highlighted annotations on thescreenshots in the report were the most useful element Whereasmost users tended to dislike the PID output included at the bottomof the report citing this information as difficult to comprehend

63 Study 3 Results Industrial ApplicabilityThe results for the applicability questions asked to 20 designersand developers who use Gvt in their daily activities is shown inFigure 7 A positive outcome for each of these statements correlatesto responses indicating that developers ldquoagreerdquo or ldquostrongly agreerdquoThe results of this study indicate a weak agreement of developersfor these statements indicating that while Gvt is generally appli-cable there are some drawbacks that prevented developers anddesigners from giving the tool unequivocal support We explorethese drawbacks by conducting semi-structured interviews

In conducting the interviews one of the authors asked the ques-tions presented in Figure 7 to 3 managers (2 from UIUX teams and1 from a Front-End development team) When asked whether Gvtcontributed to an increased quality of mobile applications at thecompany all three managers tended to agree that this was the caseFor instance one of the design managers stated ldquoCertainly yes Thetool is the industryrsquos first and the other designer manager addedldquoWhen the page is more complicated the tool is more helpful

When asked about the overall performance and accuracy of thetool in detecting DVs the manager from the implementation teamadmitted that the current detection performance of the tool is goodbut suggested that dynamic detection of some components mayimprove it stating ldquo[DVs ] can be detected pretty well [but the toolis] not very flexible For example a switch component in the designis open but the switch is off in the implementation He suggestedthat properly handling cases such as this would make the toolmore useful from a developers perspective One of the design teammanagers held a similar view stating that ldquoCurrently most errorsare layout errors so tool is accurate Static components are basicallydetected [but] maybe the next extension should focus on dynamic

ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk

components While the current version of the Gvt allows for theexclusion of regions with dynamic components it is clear that bothdesign and development teams would appreciate proper detectionof DVs for dynamic components Additionally two of the managerscommented on the ldquorigidityrdquo of the Gvtrsquos current interface andexplained that a more streamlined UI would help improve its utility

When asked about whether Gvt improved communication be-tween the design and development teams the development teammanager felt that while the tool has not improved communica-tion yet it did have the potential to do so ldquoAt present there is no[improvement] but certainly there is the potential possibility Thedesign managers generally stated that the tool has helped withcommunication particularly in clarifying subtle DVs that may havecaused arguments between teams in the past ldquoIf you consider thetime savings on discussion and arguments between the two teamsthis tool saves us a lot of time Another designer indicated that thetool is helpful at describing DVs to developers who may not be ableto recognize them with the naked eye ldquoWe found that the tool canindeed detect something that the naked eye cannot While there arecertainly further refinements that can be made to Gvt it is clearthat the tool has begun to have a positive impact of the developmentof mobile apps and as the tool evolves within the company shouldallow for continued improvements in quality and time saved

7 LIMITATIONS amp THREATS TO VALIDITYLimitations While we have illustrated thatGvt is applicable in anindustrial setting the tool is not without its limitations Currentlythe tool imposes lightweight restrictions on designers creatingSketch mock-ups chief among these being the requirement thatbounding boxes of components do not overlap Currently Gvt willtry to resolve such cases during the GUI-Comprehension stage usingan Intersection over union (IOU) metricInternal Validity While deriving the taxonomy of DVs mistakesin classification arising from subjectiveness may have introducedunexpected coding Tomitigate this threat we followed a setmethod-ology merged coding results and performed conflict resolutionConstruct Validity In our initial study (Sec 3) a threat to con-struct validity arises in the form of themanner in which coders wereexposed to presentation failures To mitigate this threat designersfrom our industrial partner manually annotated the screen pairsin order to clearly illustrate the affected GCs on the screen In ourevaluation of Gvt threats arise from our method of DV injectionusing the synthetic fault injection tool However we designed thistool to inject faults based upon both the type and distribution offaults from our DV taxonomy to mitigate this threatExternal Validity In our initial study related to the DV taxonomywe utilized a dataset from a single (albeit large) company withexamples across several different applications and screens There isthe potential that this may not generalize to other industrial mobileapplication development environments and platforms or mobile appdevelopment in general However given the relatively consistentdesign paradigms of mobile apps we expect the categories andthe sub-categories within the taxonomy to hold although it ispossible that the distribution across these categories may varyacross application development for different domains In Study 3we surveyed employees at a single (though large) company andfindings may differ in similar studies at other companies

8 RELATEDWORKWeb Presentation Failures The work most closely related to ourapproach are approaches that aim at detecting classifying and fixingpresentation failures in web applications [30ndash32 42] In comparisonto these approaches Gvt also performs detection and localizationof presentation failures but is the first to do so for mobile appsIn addition to the engineering challenges associated with buildingan approach to detect presentation failures in the mobile domain(eg collection and processing of GUI-related data) Gvt is the firstapproach to leverage metadata from software mock-up artifacts(eg Marketch) to perform GC matching based upon the spatialinformation collected from both mock-ups and dynamic applicationscreens allowing for precise detection of the different types of DVsdelineated in our industrial DV taxonomy Gvt is also the first toapply the processes of CQ CH analysis and B-PID toward detectingdifferences in the content and color of icons and images displayedin mobile apps Gvt also explicitly identifies and reports differentfaulty properties (such as errors in component location or text)Cross Browser Testing Approaches for XBT (or cross browsertesting) by Roy Choudhry et al [18 42 43] examine and automati-cally report differences in web pages rendered in multiple browsersThese approaches are currently not directly applicable to mock-updriven development for mobile appsVisualGUI Testing A concept known as Visual GUI Testing (VGT)aims to test certain visual aspects of a software applicationrsquos GUIas well as the underlying functional properties To accomplish thisvisual GUI testing usually executes actions on a target applicationsin order to exercise app functionality [13 14 23 38] In contrastto these approaches Gvt is designed to apply to mobile-specificDVs is tailored for the mock-up driven development practice andis aimed only at verifying visual properties of a mobile apprsquos GUIOther Approaches There are other approaches and techniquesthat related to identifying problems or differences with GUIs ofmobile apps Xie et al introduced GUIDE [47] a tool for GUI differ-encing between successive releases of GUIs for an app by matchingcomponents between GUI-hierarchies Gvt utilizes a matching pro-cedure for leaf node components as direct tree comparisons are notpossible in the context of mock-up driven development There hasalso been both commercial and academic work related to graphi-cal software built specifically for creating high-fidelity mobile appmock-ups or mockups that encode information for automated cre-ation of code for a target platform [4 8 9 34] However such toolstend to either impose too many restrictions on designers or do notallow for direct creation of code thus DVs still persist in practice9 CONCLUSION amp FUTUREWORKIn this paper we have formalized the problem of detecting designviolations in mobile apps and derived a taxonomy of design viola-tions based on a robust industrial dataset We presented Gvt anapproach for automatically detecting classifying and reporting de-sign violations in mobile apps and conducted a wide ranging studythat measured performance utility and industrial applicability ofthis tool Our results indicate that Gvt is effective in practice offersutility for developers and is applicable in industrial contexts

ACKNOWLEDGMENTSThe authors would like to thank Kebing Xie Roozbeh Farahbodand the developers and designers at Huawei for their support

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden

REFERENCES[1] Adobe photoshop httpwwwphotoshopcom[2] Android uiautomator httpdeveloperandroidcomtoolshelpuiautomator

indexhtml[3] Apple app store httpswwwapplecomiosapp-store[4] Fuild-ui httpswwwfluiduicom[5] Google play store httpsplaygooglecomstorehl=en[6] The marketch plugin for sketch httpsgithubcomtudou527marketch[7] Mobile apps What consumers really need and want httpsinfodynatracecom

rscompuwareimagesMobileAppSurveyReportpdf[8] Mockupio httpsmockupioabout[9] Protoio httpsprotoio[10] The sketch design tool httpswwwsketchappcom[11] Sketch extensions httpswwwsketchappcomextensions[12] Why your apprsquos ux is more important than you think httpwwwcodemagcom

Article1401041[13] E Aleacutegroth and R Feldt On the long-term use of visual gui testing in industrial

practice a case study Empirical Software Engineering 22(6)2937ndash2971 Dec 2017[14] E Alegroth Z Gao R Oliveira and A Memon Conceptualization and evaluation

of component-based testing unified with visual gui testing An empirical studyIn 2015 IEEE 8th International Conference on Software Testing Verification andValidation (ICST) pages 1ndash10 April 2015

[15] G Bavota M Linares-Vaacutesquez C Bernal-Caacuterdenas M Di Penta R Oliveto andD Poshyvanyk The impact of api change- and fault-proneness on the user ratingsof android apps Software Engineering IEEE Transactions on 41(4)384ndash407 April2015

[16] J Brooke SUS A quick and dirty usability scale In P W Jordan B WeerdmeesterA Thomas and I L Mclelland editors Usability evaluation in industry Taylorand Francis London 1996

[17] K Charmaz Constructing Grounded Theory SAGE Publications Inc 2006[18] S R Choudhary M R Prasad and A Orso Crosscheck Combining crawling and

differencing to better detect cross-browser incompatibilities in web applicationsIn Proceedings of the 2012 IEEE Fifth International Conference on Software TestingVerification and Validation ICST rsquo12 pages 171ndash180 Washington DC USA 2012IEEE Computer Society

[19] A Ciurumelea A SchaufelbAtildeČAcircijhl S Panichella and H C Gall Analyzingreviews and code of mobile apps for better release planning In 2017 IEEE 24th In-ternational Conference on Software Analysis Evolution and Reengineering (SANER)pages 91ndash102 Feb 2017

[20] A Di Sorbo S Panichella C V Alexandru J Shimagaki C A Visaggio G Can-fora and H C Gall What would users change in my app summarizing appreviews for recommending software changes In Proceedings of the 2016 24thACM SIGSOFT International Symposium on Foundations of Software EngineeringFSE 2016 pages 499ndash510 New York NY USA 2016 ACM

[21] K Holl and F Elberzhager A mobile-specific failure classification and its usageto focus quality assurance In 2014 40th EUROMICRO Conference on SoftwareEngineering and Advanced Applications pages 385ndash388 Aug 2014

[22] G Hu X Yuan Y Tang and J Yang Efficiently effectively detecting mobileapp bugs with appdoctor In Proceedings of the Ninth European Conference onComputer Systems EuroSys rsquo14 pages 181ndash1815 New York NY USA 2014ACM

[23] A Issa J Sillito and V Garousi Visual testing of graphical user interfaces Anexploratory study towards systematic definitions and approaches In 2012 14thIEEE International Symposium on Web Systems Evolution (WSE) pages 11ndash15 Sept2012

[24] N Jones Seven best practices for optimizing mobile testing efforts TechnicalReport G00248240 Gartner

[25] K Kuusinen and T Mikkonen Designing user experience for mobile apps Long-term product owner perspective In 2013 20th Asia-Pacific Software EngineeringConference volume 1 of APSECrsquo13 pages 535ndash540 Dec 2013

[26] V Lelli A Blouin and B Baudry Classifying and qualifying gui defects In 2015IEEE 8th International Conference on Software Testing Verification and Validation(ICST) pages 1ndash10 April 2015

[27] M Linares-Vaacutesquez G Bavota C Bernal-Caacuterdenas M Di Penta R Oliveto andD Poshyvanyk Api change and fault proneness A threat to the success ofandroid apps In Proceedings of the 2013 9th Joint Meeting on Foundations ofSoftware Engineering ESECFSErsquo13 pages 477ndash487 New York NY USA 2013ACM

[28] M Linares-Vaacutesquez G Bavota M D Penta R Oliveto and D Poshyvanyk Howdo API changes trigger Stack Overflow discussions a study on the android SDK

In Proceedings of the 22nd International Conference on Program ComprehensionICPCrsquo14 pages 83ndash94 2014

[29] M Linares-Vaacutesquez G Bavota M Tufano K Moran M Di Penta C VendomeC Bernal-Caacuterdenas and D Poshyvanyk Enabling mutation testing for androidapps In Proceedings of the 2017 11th Joint Meeting on Foundations of SoftwareEngineering ESECFSE 2017 pages 233ndash244 New York NY USA 2017 ACM

[30] S Mahajan A Alameer P McMinn and W G Halfond Automated repair oflayout cross browser issues using search-based techniques In InternationalConference on Software Testing and Analysis ISSTArsquo17 2017

[31] S Mahajan and W G J Halfond Detection and localization of html presentationfailures using computer vision-based techniques In Proceedings of the 8th IEEEInternational Conference on Software Testing Verification and Validation ICSTrsquo15April 2015

[32] S Mahajan B Li P Behnamghader and W G Halfond Using visual symptomsfor debugging presentation failures in web applications In Proceeding of the9th IEEE International Conference on Software Testing Verification and Validation(ICST) ICSTrsquo16 April 2016

[33] T McDonnell B Ray andM Kim An empirical study of api stability and adoptionin the android ecosystem In Proceedings of the 2013 International Conference onSoftware Maintenance ICSMrsquo13 pages 70ndash79 2013

[34] J Meskens K Luyten and K Coninx Plug-and-design Embracing mobiledevices as part of the design environment In Proceedings of the 1st ACM SIGCHISymposium on Engineering Interactive Computing Systems EICS rsquo09 pages 149ndash154 New York NY USA 2009 ACM

[35] K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk Gvt onlineappendix httpwwwandroid-dev-toolscomgvt

[36] P Morville User experience design httpsemanticstudioscomuser_experience_design

[37] B Myers Challenges of hci design and implementation interactions 1(1)73ndash83Jan 1994

[38] B N Nguyen B Robbins I Banerjee and A Memon Guitar An innovativetool for automated testing of gui-driven software Automated Software Engg21(1)65ndash105 Mar 2014

[39] T A Nguyen and C Csallner Reverse engineering mobile application userinterfaces with REMAUI In Proceedings of the 2015 30th IEEEACM Interna-tional Conference on Automated Software Engineering ASErsquo15 pages 248ndash259Washington DC USA 2015 IEEE Computer Society

[40] F PalombaM Linares-Vaacutesquez G Bavota R Oliveto M D Penta D Poshyvanykand A D Lucia User reviews matter tracking crowdsourced reviews to supportevolution of successful apps In 2015 IEEE International Conference on SoftwareMaintenance and Evolution (ICSME) pages 291ndash300 Sept 2015

[41] F Palomba P Salza A Ciurumelea S Panichella H Gall F Ferrucci andA De Lucia Recommending and localizing change requests for mobile appsbased on user reviews In Proceedings of the 39th International Conference onSoftware Engineering ICSE rsquo17 pages 106ndash117 Piscataway NJ USA 2017 IEEEPress

[42] S Roy Choudhary M R Prasad and A Orso X-pert Accurate identification ofcross-browser issues in web applications In Proceedings of the 2013 InternationalConference on Software Engineering ICSE rsquo13 pages 702ndash711 Piscataway NJUSA 2013 IEEE Press

[43] S Roy Choudhary H Versee and A Orso Webdiff Automated identification ofcross-browser issues in web applications In Proceedings of the 2010 IEEE Interna-tional Conference on Software Maintenance ICSM rsquo10 pages 1ndash10 WashingtonDC USA 2010 IEEE Computer Society

[44] T Silva da Silva A Martin F Maurer and M Silveira User-centered design andagile methods A systematic review In Proceedings of the 2011 Agile ConferenceAGILE rsquo11 pages 77ndash86 Washington DC USA 2011 IEEE Computer Society

[45] K-J Stol P Ralph and B Fitzgerald Grounded theory in software engineeringresearch A critical review and guidelines In Proceedings of the 38th InternationalConference on Software Engineering ICSE rsquo16 pages 120ndash131 New York NY USA2016 ACM

[46] A B Tucker Computer Science Handbook Second Edition Chapman amp HallCRC2004

[47] Q Xie M Grechanik C Fu and C Cumby Guide A gui differentiator In 2009IEEE International Conference on Software Maintenance ICSMrsquo09

[48] H Yee S Pattanaik and D P Greenberg Spatiotemporal sensitivity and visualattention for efficient rendering of dynamic environments ACM Trans Graph20(1)39ndash65 Jan 2001

[49] C Zeidler C Lutteroth W Stuerzlinger and G Weber Evaluating Direct Manip-ulation Operations for Constraint-Based Layout pages 513ndash529 Springer BerlinHeidelberg Berlin Heidelberg 2013

ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk

Layout Violations

39

Text Violations

23

Resource Violations

38

Horizontal Translation

10

Vertical Translation

16HampVSize5

H Size4

HampVTrans

5

Font Style Violation

8

Font Color Violation

9

Incorrect Text Violation

6

Image Color Violation

10

Missing Component12

Extra Component

6

Incorrect Image

7

Component Shape 1 Component Type 1

H = Horizontal V = VerticalTrans = Translation

Figure 2 Distribution of Different Types of Industrial DVs

31 Study Setting amp MethodologyThe goal of this study is to derive a taxonomy of the differenttypes of DVs and examine the distribution of these types inducedduring the mobile app development process The context of thisstudy is comprised of a set of 71 representative mobile app mock-up and implementation screen pairs from more than 12 differentinternal apps annotated by design teams from our industrial partnerto highlight specific instances of resolved DVs This set of screenpairs was specifically selected by the industrial design team tobe representative both in terms of diversity and distribution ofviolations that typically occur during the development process

In order to develop a taxonomy and distribution of the violationspresent in this dataset we implement an open coding methodologyconsistent with constructivist grounded theory [17] Following theadvice of recent work within the SE community [45] we stipulateour specific implementation of this type of grounded theory whilediscussing our deviations from the methods in the literature Wederived our implementation from the material discussed in [17]involving the following steps (i) establishing a research problemand questions (ii) data-collection and initial coding and (iii) focusedcoding We excluded other steps described in [17] such as memoingbecause we were building a taxonomy of labels and seeking newspecific data due to our NDA limiting the data that could be sharedThe study addressed the following research question What are thedifferent types and distributions of GUI design violations that occurduring industrial mobile app development processes

During the initial coding process three of the authors were sentthe full set of 71 screen pairs and were asked to code four pieces ofinformation for each example (i) a general category for the viola-tion (ii) a specific description of the violation (iii) the severity ofthe violation (if applicable) and (iv) the Android GC types affected(eg button) Finally we performed a second round of coding thatcombined the concepts of focused and axial coding as described in[17] During this round two of the authors merged the responsesfrom all three types of coding information where at least two ofthe three coders agreed During this phase similar coding labels

were merged (eg ldquolayout violation vs ldquospatial violation) conflictswere resolved two screen pairs were discarded due to ambiguityand cohesive categories and subcategories were formed The authoragreement for each of the four types of tags is as follows (i) generalviolation category (100) (ii) specific violation description (96)(iii) violation severity (100) and (iv) affected GC types (845)32 Grounded Theory Study ResultsOur study revealed three major categories of design violationseach with several specific subtypes We forgo detailed descriptionsand examples of violations due to space limitations but provideexamples in our online appendix [35] The derived categories andsubcategories of DVs and their distributions are illustrated in Fig 2Overall 82 DVs were identified across the 71 unique screen pairsconsidered in our study The most prevalent category of DVs in ourtaxonomy are Layout Violations (asymp 40) which concern either atranslation of a component in the x or y direction or a change in thecomponent size with translations being more common The secondmost prevalent category (asymp 36) consists of Resource Violationswhich concern missing components extra components color dif-ferences and image differences Finally about one-quarter (asymp 24)of these violations are Text Violations which concern differences incomponents that display text We observed that violations typicallyonly surfaced for ldquoleaf-level components in the GUI hierarchyThat is violations typically only affected atomic components ampnot containers or backgrounds Only 582 of examined violations(asymp 6) affected backgrounds or containers Even in these few casesthe violations also affected ldquoleaf-level components

The different types of violations correspond to different inequali-ties between the attribute tuples of corresponding GUI-componentsdefined in Sec 2 This taxonomy shows that designers are chargedwith identifying several different types of design violations a daunt-ing task particularly for hundreds of screens across several apps

4 THE GVT APPROACH41 Approach OverviewThe workflow of Gvt (Fig 3) proceeds in three stages First in theGUI-Collection Stage GUI-related information from both mock-upsand running apps is collected Next in theGUI-Comprehension Stageleaf-level GCs are parsed from the trees and a KNN-based algorithmis used to match corresponding GCs using spatial information Fi-nally in the Design Violation Detection Stage DVs are detected usinga combination of methods that leverage spatial GC information andcomputer vision techniques42 Stage 1 GUI Collection

421 Mock-Up GUI Collection Software UIUX design profes-sionals typically use professional-grade image editing software(such as Photoshop[1] or Sketch[10]) to create their mock-ups De-signers employed by our industrial partner utilize the Sketch designsoftware Sketch is popular among mobile UIUX designers due toits simple but powerful features ease of use and large library ofextensions [11] When using these tools designers often constructgraphical representations of smartphone applications by placingobjects representing GCs (which we refer to as mock-up GCs) on acanvas (representing a Screen S) that matches the typical displaysize of a target device In order to capture information encodedin these mock-ups we decided to leverage an export format that

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden3 Design Violation Detection

2 GUI-Comprehension11 Mock-Up GUI Collection

12 App Implementation GUI Collection

Resource Violation Detector

Text Violation Detector

GVT GUI Collector

apk

orapp src

Physical Deviceor Emulator

Android UIAutomator

Android adb

Screencap

AndroidApplication

Marketch Javascript

File+

+

Export Screenshot

and Marketch

Files

Mock-Up GUI Hierarchy

Implementation GUI Hierarchy

UIX Parser

GUI Hierarchy

Construction

Detection of Corresponding

Leaf Node Pairs

Identified Corresponding Component Pairs

Difference Image

Layout Violation Detector

Sketch GUI-Parser

Perceptual Image

Differencing (PID)

UIXxml File

Checksltimagegtlttextgt

Color Quantization amp Histogram Analysis

Checksltimagegt

+Missing amp Extraneous

Components

Color Quantization amp Histogram Analysis

Javascript Parser

GUIHierarchy

Construction

UIX Parser

Violation Manager

Checksltwidthgtltheighgt

Checksltxgtltygt

Design Violation Resolver

Difference Image Parser

MismatchedComponentPairs

ne

Spatial Comparator

Size Comparator

Component-LevelPerceptual Image Diff

Figure 3 Overview of GVTWorkflowwas already in use by our industrial partner an open-source Sketchextension called Marketch [6] that exports mock-ups as an html

page including a screenshot and JavaScript fileThus as input from the mock-up Gvt receives a screenshot (to

be used later in the Design Violation Detection Phase) and a directorycontaining the Marketch information The JavaScript file containsseveral pieces of information for each mock-up GC including (i) thelocation of the mock-up GC on the canvas (ii) size of the boundingbox and (iii) the textfont displayed by the mock-up GC (if any) Asshown in Figure 3- 11 we built a parser to read this informationHowever it should be noted that our approach is not tightly coupled toSketch or Marketch files1 After the Marketch files have been parsedGvt examines the extracted spatial information to build a GC hier-archy The result can be logically represented as a rooted tree whereleaf nodes contain the atomic UI-elements with which a typicaluser might interact Non-leaf node components typically representcontainers that form logical groupings of leaf node componentsand other containers In certain cases our approximation of usingmock-up GCs to represent implementation GCs may not hold Forinstance an icon which should be represented as a single GC mayconsist of several mock-up GCs representing parts of the icon Gvthandles such cases in the GUI-Comprehension stage

422 Dynamic App GUI-Collection In order to compare thethe mock-up of an app to its implementation Gvt must extractGUI-related meta-data from a running Android app Gvt is able touse Androidrsquos uiautomator framework [2] intended for UI testingto capture xml files and screenshots for a target screen of an apprunning on a physical device or emulator Each uiautomator filecontains information related to the runtime GUI-hierarchy of thetarget app including the following attributes utilized by Gvt (i)The Android component type (eg androidwidgetImageButton)(ii) the location on the screen (iii) the size of the bounding box (iv)text displayed (v) a developer assigned id The hierarchal structureof components is encoded directly in the uiautomator file and thuswe built a parser to extract GUI-hierarchy using this informationdirectly (see Fig 3- 12 )

43 Stage 2 GUI ComprehensionIn order for Gvt to find visual discrepancies between componentsexisting in the mock-up and implementation of an app it must1Similar information regarding mock-up GCs can be parsed from the html or ScalableVector Graphics (svg) format exported by other tools such as Photoshop[1]

determine which components correspond to one another Unfor-tunately the GUI-hierarchies parsed from both the Marketch anduiautomator files tend to differ dramatically due to several factorsmaking tree-based GC matching difficult First since the hierarchyconstructed using the Marketch files is generated using informationfrom the Sketch mock-up of app it is using information derivedfrom designers While designers have tremendous expertise in con-structing visual representations of apps they typically do not takethe time to construct programmatically-oriented groupings of com-ponents Furthermore designers are typically not aware of thecorrect Android component types that should be attributed to dif-ferent objects in a mock-up Second the uiautomator representationof the GUI-hierarchy contains the runtime hierarchal structure ofGCs and correct GC types This tree is typically far more com-plex containing several levels of containers grouping GCs togetherwhich is required for the responsive layouts typical of mobile apps

To overcome this challenge Gvt instead forms two collectionsof leaf-node components from both the mock-up and implementa-tion GUI-hierarchies (Fig 3- 2 ) as this information can be easilyextracted As we reported in Sec 3 the vast majority of DVs af-fects leaf-node components Once the leaf node components havebeen extracted from each hierarchy GVT employs a K-Nearest-Neighbors (KNN) algorithm utilizing a similarity function basedon the location and size of the GCs in order to perform matchingIn this setting an input leaf-node component from the mock-upwould be matched against it closest (eg K=1) neighbor from theimplementation based upon the following similarity function

γ = (|xm minus xr | + |ym minus yr | + |wm minuswr | + |hm minus hr |) (5)

Where γ is a similarity score where smaller values represent closermatches The x yw and h variables correspond to the x amp y lo-cation of the top and left-hand borders of the bounding box andthe height and width of the bounding boxes for the mock-up andimplementation GCs respectively The result is a list of GCs thatshould logically correspond to one another (corresponding GCs)

It is possible that there exist instances of missing or extraneouscomponents between the mock-up and implementation To identifythese cases our KNN algorithm employs a GC-Matching Threshold(MT ) If the similarity score of the nearest neighbor match for agiven input mock-up GC exceeds this threshold it is not matchedwith any component and will be reported as amissing GC violation

ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk

If there are unmatched GCs from the implementation they are laterreported as extraneous GC violations

Also there may be cases where a logical GC in the implemen-tation is represented as small group of mock-up GCs Gvt is ableto handle these cases using the similarity function outlined aboveFor each mock-up GC Gvt checks whether the neighboring GCsin the mockup are closer than the closest corresponding GC inthe implementation If this is the case they are merged with theprocess repeating until a logical GUI-component is represented44 Stage 3 Design Violation DetectionIn the Design Violation Detection stage of the Gvt workflow theapproach uses a combination of computer vision techniques andheuristic checking in order to effectively detect the different cate-gories of DVs derived in our taxonomy presented in Section 3

441 Perceptual Image Differencing In order to determine cor-responding GCs with visual discrepancies Gvt uses a techniquecalled Perceptual Image Differencing (PID) [48] that operates uponthe mock-up and implementation screenshots PID utilizes a modelof the human visual system to compare two images and detectvisual differences and has been used to successfully identify visualdiscrepancies in web applications in previous work [31 32] Weuse this algorithm in conjunction with the GC information derivedin the previous steps of Gvt to achieve accurate violation detec-tion For a full description of the algorithm we refer readers to[48] The PID algorithm uses several adjustable parameters includ-ing F which corresponds to the visual field of view in degrees Lwhich indicates the luminance or brightness of the image and Cwhich adjusts sensitivity to color differences The values used inour implementation are stipulated in Section 45

The output of the PID algorithm is a single difference image (Fig3- 3 ) containing difference pixels which are pixels considered to beperceptually different between the two images After processing thedifference image generated by PID Gvt extracts the implementa-tion bounding box for each corresponding pair of GCs and overlaysthe box on top of the generated difference image It then calculatesthe number of difference pixels contained within the bounding boxwhere higher numbers of difference pixels indicate potential visualdiscrepancies Thus Gvt collects all ldquosuspicious GC pairs with a of difference pixels higher than a Difference Threshold DT This setof suspicious components is then passed to the Violation Manager(Fig 3- 3 ) so that specific instances of DVs can be detected

442 Detecting Layout Violations The first general category ofDVs thatGvt detects are Layout Violations According the taxonomyderived in Sec 3 there are six specific layout DV categories thatrelate to two component properties (i) screen location (ie ltxygtposition) and (ii) size (ie lthwgt of the GC bounding box) Gvt firstchecks for the three types of translationDVs utilizing a heuristic thatmeasures the distance from the top and left-hand edges of matchedcomponents If the difference between the components in either thex ory dimension is greater than a Layout Threshold (LT ) then thesecomponents are reported as a Layout DV Using the LT avoids triviallocation discrepancies within design tolerances being reported asviolations and can be set by a designer or developer using the toolWhen detecting the three types of size DVs in the derived designviolation taxonomyGvt utilizes a heuristic that compares thewidthand height of the bounding boxes of corresponding components If

the width or height of the bounding boxes differ by more than theLT then a layout violation is reported

443 Detecting Text Violations The next general type of DVthat Gvt detects are Text Violations of which there are three spe-cific types (i) Font Color (ii) Font Style and (iii) Incorrect TextContent These detection strategies are only applied to pairs oftext-based components as determined by uiautomator informationTo detect font color violations Gvt extracts cropped images foreach pair of suspicious text components by cropping the mock-upand implementation screenshots according to the componentrsquos re-spective bounding boxes Next Color Quantization (CQ) is appliedto accumulate instances of all unique RGB values expressed in thecomponent-specific images This quantization information is thenused to construct a Color Histogram (CH) (Fig 3- 3 ) Gvt computesthe normalized Euclidean distance between the extracted ColorHistograms for the corresponding GC pairs and if the Histogramsdo not match within a Color Threshold (CT) then a Font-Color DVis reported and the top-3 colors (ie centroids) from each CH arerecorded in theGvt report Likewise if the colors domatch then thePID discrepancy identified earlier is due to the Font-Style changing(provided no existing layout DVs) and thus a Font-Style Violationis reported Finally to detect incorrect text content Gvt utilizesthe textual information preprocessed to remove whitespace andnormalize letter cases and performs a string comparison If thestrings do not match then an Incorrect Text Content DV is reported

444 Detecting Resource Violations Gvt is able to detect thefollowing resource DVs (i) missing component (ii) extraneous com-ponent (iii) image color (iv) incorrect images and (v) componentshape The detection and distinction between Incorrect Image DVsand Image Color DVs requires an analysis that combines two differ-ent computer vision techniques To perform this analysis croppedimages from the mock-up and implementation screenshots accord-ing to corresponding GCs respective bounding boxes are extractedThe goal of this analysis is to determine when the content of image-based GCs differ as opposed to only the colors of the GCs differingTo accomplish this Gvt leverages PID applied to extracted GCimages converted to a binary color space (B-PID) in order to detectdifferences in content and CQ and CH analysis to determine differ-ences in color (Sec 443) To perform the B-PID procedure croppedGC images are converted to a binary color space by extracting pixelintensities and then applying a binary transformation to the inten-sity values (eg converting the images to intensity independentblack amp white) Then PID is run on the color-neutral version ofthese images If the images differ by more than an Image DifferenceThreshold (IDT ) then an Incorrect Image DV (which encompassesthe Component Shape DV ) is reported If the component passes thebinary PID check then Gvt utilizes the same CQ and CH process-ing technique described above to detect image color DVs Missingand extraneous components are detected as described in Sec 43

445 Generating Violation Reports In order to provide develop-ers and designers with effective information regarding the detectedDVs Gvt generates an html report that for each detected viola-tion contains the following (i) a natural language description ofthe design violation(s) (ii) an annotated screenshot of the app im-plementation with the affected GUI-component highlighted (iii)cropped screenshots of the affected GCs from both the design and

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden

implementation screenshots (iv) links to affected lines of applica-tion source code (v) color information extracted from the CH forGCs identified to have color mismatches and (vi) the differenceimage generated by PID The source code links are generated bymatching the ids extracted from the uiautomator information backto their declarations in the layout xml files in the source code (egthose located in the res directory of an apprsquos source code) Weprovide examples of generated reports in our online appendix [35]45 Implementation amp Industrial CollaborationOur implementation of Gvt was developed in Java with a SwingGUI In addition to running the Gvt analysis the tool executableallows for one-click capture of uiautomator files and screenshotsfrom a connected device or emulator Several acceptance tests ofmock-upimplementation screen pairs with pre-existing violationsfrom apps under development within our industrial partner wereused to guide the development of the tool 12 Periodic releases ofbinaries for bothWindows and Mac were made to deploy the tool todesigners and developers within the company The authors of thispaper held regular bi-weekly meetings with members of the designand development teams to plan features and collect feedback

Using the acceptance tests and feedback from our collaboratorswe tuned the various thresholds and parameters of the tool for bestperformance The PID algorithm settings were tuned for sensitivityto capture subtle visual inconsistencies which are then later filteredthrough additional CV techniques F was set to 45 L was set to100cdm2 and C was set to 1 The GC-Matching Threshold (MC)was set to 18th the screen width of a target device the DT fordetermining suspicious GCs was set to 20 The LT was set to 5pixels (based on designer preference) the CT which determinesthe degree to which colors must match for color-based DVs was setto 85 and finally the IDT was set to 20 Gvt allows for a userto change these settings if desired additionally users are capableof defining areas of dynamic content (eg loaded from networkactivity) which should be ignored by the Gvt analysis5 DESIGN OF THE EXPERIMENTSTo evaluate Gvtrsquos performance usefulness and applicability weperform three complimentary studies answering the following RQs

bull RQ1 How well does Gvt perform in terms of detecting andclassifying design violations

bull RQ2 What utility can Gvt provide from the viewpoint ofAndroid developers

bull RQ3What is the industrial applicability of Gvt in terms ofimproving the mobile application development workflow

RQ1 and RQ2 focus on quantitatively measuring the performanceof Gvt and the utility it provides to developers through a controlledempirical and a user study respectively RQ3 reports the results ofa survey and semi-structured interviews with our collaboratorsaimed at investigating the industrial applicability of Gvt51 Study 1 Gvt Effectiveness amp PerformanceThe goal of the first study is to quantitatively measure Gvt in termsof its precision and recall in both detecting and classifying DVs

511 Study Context To carry out a controlled quantitativestudy we manually reverse engineered Sketch mockups for tenscreens for eight of the most popular apps on Google Play To de-rive this set we downloaded the top-10 apps from each category on

the Google-Play store removing the various categories correspond-ing to games (as these have non-standard GUI-components thatGvt does not support) We then randomly sampled one app fromeach of the remaining 33 categories eliminating duplicates (sinceapps can belong to more than one category) We then manuallycollected screenshots and uiautomator files from two screens foreach application using a Nexus 5 attempting to capture the ldquomainrdquoscreen that a user would typically interact with and one secondaryscreen Using the uiautomator files we generated cropped screen-shots of all the leaf nodes components for each screen of the appFrom these we were able generate 10 screens from 8 applicationsthat successfully ran through Gvt without any reported violations

512 Synthetic DV Injection With a set of correct mock-ups cor-responding to implementation screens in an app we needed a suit-able method to introduce DVs into our subjects To this end we con-structed a synthetic DV injection tool that modifies the uiautomator

xml files and corresponding screenshots in order to introduce de-sign violations from our taxonomy presented in Sec 3 The toolis composed of two components (i) an XML Parser that reads andextracts components from the screen then (ii) a Violation Generatorthat randomly selects components and injects synthetic violationsWe implemented injection for the following types of DVsLocation Violation The component is moved either horizontallyvertically or in both directions within the same container Howeverthemaximum distance from the original point is limited by a quarterof the width of the screen size This was based on the severity ofLayout Violations in our study described in Section 3 In order togenerate the image we cropped the component and moved it to thenew location replacing all the original pixels by the most prominentcolor from the surroundings in the original locationSize Violation The component size either increases or decreasesby 20 of the original size For instances where the component sizedecreases we replaced all the pixels by the most prominent colorfrom the surroundings of the original sizeMissing Component Violation This violation removes a leafcomponent from the screen replacing the original pixels by themost prominent color from the surrounding backgroundImage Violation We perturb 40 of the pixels in an image byrandomly generating an RGB value for the pixels affectedImage Color Violation This rule perturbs the color of an imageby shifting the hue of image colors by 30degComponent Color Violation This uses the same process as forImage Color Violations but we change the color by 180degFont Violation This violation randomly selects a font from theset of Arial Comic Sans MS Courier Roboto or Times Roman andapplies it to a TextView componentFont Color Violation changes the text color of a TextView com-ponent We extracted the text color using CH analysis then wechanged the color using same strategy as for Image Color Violations

513 Study Methodology In injecting the synthetic faults wetook several measures to simulate the creation of realistic faultsFirst we delineated 200 different types of design violations accord-ing to the distribution defined in our DV taxonomy in Sec 3 Wethen created a pool of 100 screens by creating random copies ofthe both the uiautomator xml files and screenshots from our initialset of 10 screens We then used the synthetic DV injection tool toseed faults into the pool of 100 screens according to the following

ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk

criteria (i) No screen can contain more than 3 injected DVs (ii)each GC should have a maximum of 1 DV injected and (iii) Eachscreen must have at least 1 injected DV After the DVs were seededeach of the 100 screens and 200 DVs were manually inspected forcorrectness Due to the random nature of the tool a small num-ber of erroneous DVs were excluded and regenerated during thisprocess (eg color perturbed to perceptually similar color) Thebreakdown of injected DVs is shown in Figure 4 and the full datasetwith description is included in our online appendix [35]

Once the final set of screens with injected violations was derivedwe ran Gvt across these subjects and measured four metrics (i)detection precision (DP ) (ii) classification precision (CP ) (iii) recall(R) and (iv) execution time per screen (ET ) We make a distinctionbetween detection and classification in our dataset because it is pos-sible thatGvt is capable of detecting but misclassifying a particularDV (eg an image color DV misclassified as an incorrect image DV )DP CP and R were measured according to the following formulas

DP CP =Tp

Tp + FpR =

Tp

Tp + Fn(6)

where for DP Tp represent injected design violations that weredetected and for CP Tp represents injected violations that wereboth detected and classified correctly In each case Fp correspondto detected DVs that were either not injected or misclassified ForRecallTp represents injected violations that were correctly detectedand Fn represents injected violations that were not detected Tocollect these measures two authors manually examined the reportsfrom Gvt in order to collect the metrics

52 Study 2 Gvt UtilitySince the ultimate goal of an approach like Gvt is to improvethe workflow of developers the goal of this second study is tomeasure the utility (ie benefit) that Gvt provides to developersby investigating two phenomena (i) The accuracy and effort ofdevelopers in detecting and classifying DVs and (ii) the perceivedutility of Gvt reports in helping to identify and resolve DVs

521 Study Context We randomly derived two sets of screensto investigate the two phenomena outlined above First we ran-domly sampled two mutually exclusive sets of 25 and 20 screensrespectively from the 100 used in Study 1 ensuring at least oneinstance of each type of DV was included in the set This resulted inboth sets of screens containing 40 design violations in total The cor-rect mockup screenshot corresponding to each screen sampled fromthe study were also extracted creating pairs of ldquocorrect mockupand ldquoincorrect implementation screenshots 10 participants withat least 5 years of Android development experience were contactedvia email to participate in the survey

522 Study Methodology We created an online survey withfour sections In the first section participants were given back-ground information regarding the definition of DVs and the differ-ent types of DVs derived in our taxonomy In the second sectionparticipants were asked about demographic information such asprogramming experience and education level In the third sectioneach participant was exposed to 5 mock-up implementation screenpairs (displayed side by side on the survey web page) and asked toidentify any observed design violations Descriptions of the DVswere given at the top of this page for reference For each screenpair participants were presented with a dropdown menu to select

80

85

90

95

100

Font

Colo

r(18)

Font

Styl

e(18

)

Imag

e Colo

r(22)

Imag

e(16

)

Miss

ing G

C(26)

Size(1

8)

Text

Conte

nt(1

4)

Trans

lation

(68)

DV Type

Per

cent

age

MetricCPDPR

Figure 4 Study 1 - Detection Precision (DP ) ClassificationPrecision (CP ) and Recall (R)a type for an observed DV and a text field to describe the errorin more detail For each participant one of the 5 mock-up screenswas a control containing no injected violations The 25 screenswere assigned to participants such that each screen was observedby two participants and the order of the screens presented to eachparticipant was randomized to avoid bias To measure the effective-ness of participants in detecting and describing DVs we leveragethe DP CP and R metrics introduced in Study 1 In the fourth sec-tion participants were presented with two screen pairs from thesecond set of 20 sampled from the user study as well as the Gvtreports for these screens Participants were then asked to answer5 user-preferences (UP) and 5 user experience (UX) questions aboutthese reports which are presented in the following section The UPquestions were developed according to the user experience hon-eycomb originally developed by Morville [36] and were posed toparticipants as free form text entry questions We forgo a discussionof the free-form question responses due to space limitations butwe offer full anonymized participant responses in our online ap-pendix [35] We derived the Likert scale-based UX questions usingthe SUS usability scale by Brooke [16]

53 Study 3 Industrial Applicability of GvtThe goal of this final study is determine industrial applicability ofGvt To investigate this we worked with Huawei to collect twosources of information (i) the results of a survey sent to designersand developers who used Gvt in their daily developmentdesignworkflow and (ii) semi-structured interviews with both design anddevelopment managers whose teams have adopted the use of Gvt

531 Study Context ampMethodology We created a survey posingquestions related to the applicability of Gvt to industrial designersand developers These questions are shown in Fig 7 The semi-structured interviews were conducted in Chinese recorded andthen later translated During the interview managers were askedto respond to four questions related to the impact and performanceof the tool in practice We include discussions of the responses inSection 6 and stipulate full questions in our appendix

6 EMPIRICAL RESULTS61 Study 1 Results GVT PerformanceThe results of Study 1 are shown in Figure 4 This figure shows theaverageDP CP and R for each type of seeded violation over the 200seeded faults and the number of faults seeded into each category(following the distributions of our derived taxonomy) are shown onthe x-axis Overall these results are extremely encouraging withthe overall DP achieving 994 the average CP being 984 and

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden

CPDP

R

000 025 050 075 100

Figure 5 Study 2 - Developer CP DP and R

I found these reports unnecessarily complex

I found these reports very cumbersome to read

I think that I would like to use these reports frequently for Identifying Presentation Issues

I thought these reports were very useful for accurately identifying Presentation Errors

These reports are easy to readunderstand

SD D N A SA

Figure 6 Study 2 - UXQuestion Responses SD=Strongly Dis-agree D=Disagree N=Neutral A=Agree SA=Strongly Agree

the average R reaching 965 This illustrates that Gvt is capableof detecting seeded faults designed to emulate both the type anddistribution of DVs encountered in industrial settings While Gvtachieved at least 85 precision for each type of seeded DV it per-formed worse on some types of violations compared to others Forinstance Gvt saw its lowest precision values for the Font-Style andFont-Color violations typically due to the fact that the magnitudeof perturbation for the color or font type was not large enough tosurpass the Color or Image Difference Thresholds (CT amp IDT ) Gvttook 368 mins to process and generate reports for the set of 100screens with injected DVs or 22 sec per screen pair This executioncost was generally acceptable by our industrial collaborators

62 Study 2 Results GVT UtilityThe DP CP and R results representing the Android developersability to correctly detect and classify DVs is shown in Figure 5as box-plots across all 10 participants Here we found CP=DP aswhen a user misclassified violations they also did not detect themAs this figure shows the Android developers generally performedmuch worse compared to Gvt achieving an average CP of underasymp 60 and an average R of asymp 50 The sources of this performanceloss for the study participants compared to Gvt was fourfold (i)participants tended to report minor acceptable differences in fontsacross the examples (despite the instructions clearly stating notto report such violations) (ii) users tended to attribute more thanone DV to a single component specifically for font style and fontcolor violations despite instructions to report only one (iii) userstended to misclassify DVs based on the provided categories (egclassifying a layout DV for a Text GC as an incorrect text DV ) and(iv) participants missed reporting many of the injected DVs lead-ing to the low recall numbers These results indicate that at thevery least developers can struggle to both detect and classify DVsbetween mock-up and implementation screen pairs signaling theneed for an automated system to check for DVs before implementedapps are sent to a UIUX team for auditing This result confirmsthe notion that developers may not be as sensitive to small DVsin the GUI as the designers who created the GUI specificationsFurthermore this finding is notable because as part of the iterativeprocess of resolving design violations designers must communicateto developers DVs and developers must recognize and understandthese DVs in order to properly resolve them This process is oftencomplicated due to ambiguous descriptions of DVs from designers

The GVT allowed for better transfer of the design from mockminusups to the implementation of the app

The GVT has helped you to reduce the time required for verifying design violations

The GVT is able to accurately report existing design violations in productionminusquality applications

The GVT tool helped my team (designimplementation) communicate with other teams (implementationdesign)

regarding GUI design violations

Using the GUIminusVerification tool (GVT) helped to improve the quality of mobile

applications produced by industrial partner

SD D N A SA

Figure 7 Study 3 - Applicability Questions SD=Strongly Dis-agree D=Disagree N=Neutral A=Agree SA=Strongly Agree

to developers or developers disagreeing with designers over the ex-istence or type of a DV In contrast to this fragmented process Gvtprovides clear unambiguous reports that facilitate communicationbetween designers and developers

Figure 6 illustrates the responses to the likert based UX ques-tions and the results are quite encouraging In general participantsfound that the reports from Gvt were easy to read useful for iden-tifying DVs and indicated that they would like to use the reports foridentifying DVs Participants also indicated that the reports werenot unnecessarily complex or difficult to read We asked the partic-ipants about their preferences for the Gvt reports as well askingabout the most and least useful information in the reports Everysingle participant indicated that the highlighted annotations on thescreenshots in the report were the most useful element Whereasmost users tended to dislike the PID output included at the bottomof the report citing this information as difficult to comprehend

63 Study 3 Results Industrial ApplicabilityThe results for the applicability questions asked to 20 designersand developers who use Gvt in their daily activities is shown inFigure 7 A positive outcome for each of these statements correlatesto responses indicating that developers ldquoagreerdquo or ldquostrongly agreerdquoThe results of this study indicate a weak agreement of developersfor these statements indicating that while Gvt is generally appli-cable there are some drawbacks that prevented developers anddesigners from giving the tool unequivocal support We explorethese drawbacks by conducting semi-structured interviews

In conducting the interviews one of the authors asked the ques-tions presented in Figure 7 to 3 managers (2 from UIUX teams and1 from a Front-End development team) When asked whether Gvtcontributed to an increased quality of mobile applications at thecompany all three managers tended to agree that this was the caseFor instance one of the design managers stated ldquoCertainly yes Thetool is the industryrsquos first and the other designer manager addedldquoWhen the page is more complicated the tool is more helpful

When asked about the overall performance and accuracy of thetool in detecting DVs the manager from the implementation teamadmitted that the current detection performance of the tool is goodbut suggested that dynamic detection of some components mayimprove it stating ldquo[DVs ] can be detected pretty well [but the toolis] not very flexible For example a switch component in the designis open but the switch is off in the implementation He suggestedthat properly handling cases such as this would make the toolmore useful from a developers perspective One of the design teammanagers held a similar view stating that ldquoCurrently most errorsare layout errors so tool is accurate Static components are basicallydetected [but] maybe the next extension should focus on dynamic

ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk

components While the current version of the Gvt allows for theexclusion of regions with dynamic components it is clear that bothdesign and development teams would appreciate proper detectionof DVs for dynamic components Additionally two of the managerscommented on the ldquorigidityrdquo of the Gvtrsquos current interface andexplained that a more streamlined UI would help improve its utility

When asked about whether Gvt improved communication be-tween the design and development teams the development teammanager felt that while the tool has not improved communica-tion yet it did have the potential to do so ldquoAt present there is no[improvement] but certainly there is the potential possibility Thedesign managers generally stated that the tool has helped withcommunication particularly in clarifying subtle DVs that may havecaused arguments between teams in the past ldquoIf you consider thetime savings on discussion and arguments between the two teamsthis tool saves us a lot of time Another designer indicated that thetool is helpful at describing DVs to developers who may not be ableto recognize them with the naked eye ldquoWe found that the tool canindeed detect something that the naked eye cannot While there arecertainly further refinements that can be made to Gvt it is clearthat the tool has begun to have a positive impact of the developmentof mobile apps and as the tool evolves within the company shouldallow for continued improvements in quality and time saved

7 LIMITATIONS amp THREATS TO VALIDITYLimitations While we have illustrated thatGvt is applicable in anindustrial setting the tool is not without its limitations Currentlythe tool imposes lightweight restrictions on designers creatingSketch mock-ups chief among these being the requirement thatbounding boxes of components do not overlap Currently Gvt willtry to resolve such cases during the GUI-Comprehension stage usingan Intersection over union (IOU) metricInternal Validity While deriving the taxonomy of DVs mistakesin classification arising from subjectiveness may have introducedunexpected coding Tomitigate this threat we followed a setmethod-ology merged coding results and performed conflict resolutionConstruct Validity In our initial study (Sec 3) a threat to con-struct validity arises in the form of themanner in which coders wereexposed to presentation failures To mitigate this threat designersfrom our industrial partner manually annotated the screen pairsin order to clearly illustrate the affected GCs on the screen In ourevaluation of Gvt threats arise from our method of DV injectionusing the synthetic fault injection tool However we designed thistool to inject faults based upon both the type and distribution offaults from our DV taxonomy to mitigate this threatExternal Validity In our initial study related to the DV taxonomywe utilized a dataset from a single (albeit large) company withexamples across several different applications and screens There isthe potential that this may not generalize to other industrial mobileapplication development environments and platforms or mobile appdevelopment in general However given the relatively consistentdesign paradigms of mobile apps we expect the categories andthe sub-categories within the taxonomy to hold although it ispossible that the distribution across these categories may varyacross application development for different domains In Study 3we surveyed employees at a single (though large) company andfindings may differ in similar studies at other companies

8 RELATEDWORKWeb Presentation Failures The work most closely related to ourapproach are approaches that aim at detecting classifying and fixingpresentation failures in web applications [30ndash32 42] In comparisonto these approaches Gvt also performs detection and localizationof presentation failures but is the first to do so for mobile appsIn addition to the engineering challenges associated with buildingan approach to detect presentation failures in the mobile domain(eg collection and processing of GUI-related data) Gvt is the firstapproach to leverage metadata from software mock-up artifacts(eg Marketch) to perform GC matching based upon the spatialinformation collected from both mock-ups and dynamic applicationscreens allowing for precise detection of the different types of DVsdelineated in our industrial DV taxonomy Gvt is also the first toapply the processes of CQ CH analysis and B-PID toward detectingdifferences in the content and color of icons and images displayedin mobile apps Gvt also explicitly identifies and reports differentfaulty properties (such as errors in component location or text)Cross Browser Testing Approaches for XBT (or cross browsertesting) by Roy Choudhry et al [18 42 43] examine and automati-cally report differences in web pages rendered in multiple browsersThese approaches are currently not directly applicable to mock-updriven development for mobile appsVisualGUI Testing A concept known as Visual GUI Testing (VGT)aims to test certain visual aspects of a software applicationrsquos GUIas well as the underlying functional properties To accomplish thisvisual GUI testing usually executes actions on a target applicationsin order to exercise app functionality [13 14 23 38] In contrastto these approaches Gvt is designed to apply to mobile-specificDVs is tailored for the mock-up driven development practice andis aimed only at verifying visual properties of a mobile apprsquos GUIOther Approaches There are other approaches and techniquesthat related to identifying problems or differences with GUIs ofmobile apps Xie et al introduced GUIDE [47] a tool for GUI differ-encing between successive releases of GUIs for an app by matchingcomponents between GUI-hierarchies Gvt utilizes a matching pro-cedure for leaf node components as direct tree comparisons are notpossible in the context of mock-up driven development There hasalso been both commercial and academic work related to graphi-cal software built specifically for creating high-fidelity mobile appmock-ups or mockups that encode information for automated cre-ation of code for a target platform [4 8 9 34] However such toolstend to either impose too many restrictions on designers or do notallow for direct creation of code thus DVs still persist in practice9 CONCLUSION amp FUTUREWORKIn this paper we have formalized the problem of detecting designviolations in mobile apps and derived a taxonomy of design viola-tions based on a robust industrial dataset We presented Gvt anapproach for automatically detecting classifying and reporting de-sign violations in mobile apps and conducted a wide ranging studythat measured performance utility and industrial applicability ofthis tool Our results indicate that Gvt is effective in practice offersutility for developers and is applicable in industrial contexts

ACKNOWLEDGMENTSThe authors would like to thank Kebing Xie Roozbeh Farahbodand the developers and designers at Huawei for their support

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden

REFERENCES[1] Adobe photoshop httpwwwphotoshopcom[2] Android uiautomator httpdeveloperandroidcomtoolshelpuiautomator

indexhtml[3] Apple app store httpswwwapplecomiosapp-store[4] Fuild-ui httpswwwfluiduicom[5] Google play store httpsplaygooglecomstorehl=en[6] The marketch plugin for sketch httpsgithubcomtudou527marketch[7] Mobile apps What consumers really need and want httpsinfodynatracecom

rscompuwareimagesMobileAppSurveyReportpdf[8] Mockupio httpsmockupioabout[9] Protoio httpsprotoio[10] The sketch design tool httpswwwsketchappcom[11] Sketch extensions httpswwwsketchappcomextensions[12] Why your apprsquos ux is more important than you think httpwwwcodemagcom

Article1401041[13] E Aleacutegroth and R Feldt On the long-term use of visual gui testing in industrial

practice a case study Empirical Software Engineering 22(6)2937ndash2971 Dec 2017[14] E Alegroth Z Gao R Oliveira and A Memon Conceptualization and evaluation

of component-based testing unified with visual gui testing An empirical studyIn 2015 IEEE 8th International Conference on Software Testing Verification andValidation (ICST) pages 1ndash10 April 2015

[15] G Bavota M Linares-Vaacutesquez C Bernal-Caacuterdenas M Di Penta R Oliveto andD Poshyvanyk The impact of api change- and fault-proneness on the user ratingsof android apps Software Engineering IEEE Transactions on 41(4)384ndash407 April2015

[16] J Brooke SUS A quick and dirty usability scale In P W Jordan B WeerdmeesterA Thomas and I L Mclelland editors Usability evaluation in industry Taylorand Francis London 1996

[17] K Charmaz Constructing Grounded Theory SAGE Publications Inc 2006[18] S R Choudhary M R Prasad and A Orso Crosscheck Combining crawling and

differencing to better detect cross-browser incompatibilities in web applicationsIn Proceedings of the 2012 IEEE Fifth International Conference on Software TestingVerification and Validation ICST rsquo12 pages 171ndash180 Washington DC USA 2012IEEE Computer Society

[19] A Ciurumelea A SchaufelbAtildeČAcircijhl S Panichella and H C Gall Analyzingreviews and code of mobile apps for better release planning In 2017 IEEE 24th In-ternational Conference on Software Analysis Evolution and Reengineering (SANER)pages 91ndash102 Feb 2017

[20] A Di Sorbo S Panichella C V Alexandru J Shimagaki C A Visaggio G Can-fora and H C Gall What would users change in my app summarizing appreviews for recommending software changes In Proceedings of the 2016 24thACM SIGSOFT International Symposium on Foundations of Software EngineeringFSE 2016 pages 499ndash510 New York NY USA 2016 ACM

[21] K Holl and F Elberzhager A mobile-specific failure classification and its usageto focus quality assurance In 2014 40th EUROMICRO Conference on SoftwareEngineering and Advanced Applications pages 385ndash388 Aug 2014

[22] G Hu X Yuan Y Tang and J Yang Efficiently effectively detecting mobileapp bugs with appdoctor In Proceedings of the Ninth European Conference onComputer Systems EuroSys rsquo14 pages 181ndash1815 New York NY USA 2014ACM

[23] A Issa J Sillito and V Garousi Visual testing of graphical user interfaces Anexploratory study towards systematic definitions and approaches In 2012 14thIEEE International Symposium on Web Systems Evolution (WSE) pages 11ndash15 Sept2012

[24] N Jones Seven best practices for optimizing mobile testing efforts TechnicalReport G00248240 Gartner

[25] K Kuusinen and T Mikkonen Designing user experience for mobile apps Long-term product owner perspective In 2013 20th Asia-Pacific Software EngineeringConference volume 1 of APSECrsquo13 pages 535ndash540 Dec 2013

[26] V Lelli A Blouin and B Baudry Classifying and qualifying gui defects In 2015IEEE 8th International Conference on Software Testing Verification and Validation(ICST) pages 1ndash10 April 2015

[27] M Linares-Vaacutesquez G Bavota C Bernal-Caacuterdenas M Di Penta R Oliveto andD Poshyvanyk Api change and fault proneness A threat to the success ofandroid apps In Proceedings of the 2013 9th Joint Meeting on Foundations ofSoftware Engineering ESECFSErsquo13 pages 477ndash487 New York NY USA 2013ACM

[28] M Linares-Vaacutesquez G Bavota M D Penta R Oliveto and D Poshyvanyk Howdo API changes trigger Stack Overflow discussions a study on the android SDK

In Proceedings of the 22nd International Conference on Program ComprehensionICPCrsquo14 pages 83ndash94 2014

[29] M Linares-Vaacutesquez G Bavota M Tufano K Moran M Di Penta C VendomeC Bernal-Caacuterdenas and D Poshyvanyk Enabling mutation testing for androidapps In Proceedings of the 2017 11th Joint Meeting on Foundations of SoftwareEngineering ESECFSE 2017 pages 233ndash244 New York NY USA 2017 ACM

[30] S Mahajan A Alameer P McMinn and W G Halfond Automated repair oflayout cross browser issues using search-based techniques In InternationalConference on Software Testing and Analysis ISSTArsquo17 2017

[31] S Mahajan and W G J Halfond Detection and localization of html presentationfailures using computer vision-based techniques In Proceedings of the 8th IEEEInternational Conference on Software Testing Verification and Validation ICSTrsquo15April 2015

[32] S Mahajan B Li P Behnamghader and W G Halfond Using visual symptomsfor debugging presentation failures in web applications In Proceeding of the9th IEEE International Conference on Software Testing Verification and Validation(ICST) ICSTrsquo16 April 2016

[33] T McDonnell B Ray andM Kim An empirical study of api stability and adoptionin the android ecosystem In Proceedings of the 2013 International Conference onSoftware Maintenance ICSMrsquo13 pages 70ndash79 2013

[34] J Meskens K Luyten and K Coninx Plug-and-design Embracing mobiledevices as part of the design environment In Proceedings of the 1st ACM SIGCHISymposium on Engineering Interactive Computing Systems EICS rsquo09 pages 149ndash154 New York NY USA 2009 ACM

[35] K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk Gvt onlineappendix httpwwwandroid-dev-toolscomgvt

[36] P Morville User experience design httpsemanticstudioscomuser_experience_design

[37] B Myers Challenges of hci design and implementation interactions 1(1)73ndash83Jan 1994

[38] B N Nguyen B Robbins I Banerjee and A Memon Guitar An innovativetool for automated testing of gui-driven software Automated Software Engg21(1)65ndash105 Mar 2014

[39] T A Nguyen and C Csallner Reverse engineering mobile application userinterfaces with REMAUI In Proceedings of the 2015 30th IEEEACM Interna-tional Conference on Automated Software Engineering ASErsquo15 pages 248ndash259Washington DC USA 2015 IEEE Computer Society

[40] F PalombaM Linares-Vaacutesquez G Bavota R Oliveto M D Penta D Poshyvanykand A D Lucia User reviews matter tracking crowdsourced reviews to supportevolution of successful apps In 2015 IEEE International Conference on SoftwareMaintenance and Evolution (ICSME) pages 291ndash300 Sept 2015

[41] F Palomba P Salza A Ciurumelea S Panichella H Gall F Ferrucci andA De Lucia Recommending and localizing change requests for mobile appsbased on user reviews In Proceedings of the 39th International Conference onSoftware Engineering ICSE rsquo17 pages 106ndash117 Piscataway NJ USA 2017 IEEEPress

[42] S Roy Choudhary M R Prasad and A Orso X-pert Accurate identification ofcross-browser issues in web applications In Proceedings of the 2013 InternationalConference on Software Engineering ICSE rsquo13 pages 702ndash711 Piscataway NJUSA 2013 IEEE Press

[43] S Roy Choudhary H Versee and A Orso Webdiff Automated identification ofcross-browser issues in web applications In Proceedings of the 2010 IEEE Interna-tional Conference on Software Maintenance ICSM rsquo10 pages 1ndash10 WashingtonDC USA 2010 IEEE Computer Society

[44] T Silva da Silva A Martin F Maurer and M Silveira User-centered design andagile methods A systematic review In Proceedings of the 2011 Agile ConferenceAGILE rsquo11 pages 77ndash86 Washington DC USA 2011 IEEE Computer Society

[45] K-J Stol P Ralph and B Fitzgerald Grounded theory in software engineeringresearch A critical review and guidelines In Proceedings of the 38th InternationalConference on Software Engineering ICSE rsquo16 pages 120ndash131 New York NY USA2016 ACM

[46] A B Tucker Computer Science Handbook Second Edition Chapman amp HallCRC2004

[47] Q Xie M Grechanik C Fu and C Cumby Guide A gui differentiator In 2009IEEE International Conference on Software Maintenance ICSMrsquo09

[48] H Yee S Pattanaik and D P Greenberg Spatiotemporal sensitivity and visualattention for efficient rendering of dynamic environments ACM Trans Graph20(1)39ndash65 Jan 2001

[49] C Zeidler C Lutteroth W Stuerzlinger and G Weber Evaluating Direct Manip-ulation Operations for Constraint-Based Layout pages 513ndash529 Springer BerlinHeidelberg Berlin Heidelberg 2013

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden3 Design Violation Detection

2 GUI-Comprehension11 Mock-Up GUI Collection

12 App Implementation GUI Collection

Resource Violation Detector

Text Violation Detector

GVT GUI Collector

apk

orapp src

Physical Deviceor Emulator

Android UIAutomator

Android adb

Screencap

AndroidApplication

Marketch Javascript

File+

+

Export Screenshot

and Marketch

Files

Mock-Up GUI Hierarchy

Implementation GUI Hierarchy

UIX Parser

GUI Hierarchy

Construction

Detection of Corresponding

Leaf Node Pairs

Identified Corresponding Component Pairs

Difference Image

Layout Violation Detector

Sketch GUI-Parser

Perceptual Image

Differencing (PID)

UIXxml File

Checksltimagegtlttextgt

Color Quantization amp Histogram Analysis

Checksltimagegt

+Missing amp Extraneous

Components

Color Quantization amp Histogram Analysis

Javascript Parser

GUIHierarchy

Construction

UIX Parser

Violation Manager

Checksltwidthgtltheighgt

Checksltxgtltygt

Design Violation Resolver

Difference Image Parser

MismatchedComponentPairs

ne

Spatial Comparator

Size Comparator

Component-LevelPerceptual Image Diff

Figure 3 Overview of GVTWorkflowwas already in use by our industrial partner an open-source Sketchextension called Marketch [6] that exports mock-ups as an html

page including a screenshot and JavaScript fileThus as input from the mock-up Gvt receives a screenshot (to

be used later in the Design Violation Detection Phase) and a directorycontaining the Marketch information The JavaScript file containsseveral pieces of information for each mock-up GC including (i) thelocation of the mock-up GC on the canvas (ii) size of the boundingbox and (iii) the textfont displayed by the mock-up GC (if any) Asshown in Figure 3- 11 we built a parser to read this informationHowever it should be noted that our approach is not tightly coupled toSketch or Marketch files1 After the Marketch files have been parsedGvt examines the extracted spatial information to build a GC hier-archy The result can be logically represented as a rooted tree whereleaf nodes contain the atomic UI-elements with which a typicaluser might interact Non-leaf node components typically representcontainers that form logical groupings of leaf node componentsand other containers In certain cases our approximation of usingmock-up GCs to represent implementation GCs may not hold Forinstance an icon which should be represented as a single GC mayconsist of several mock-up GCs representing parts of the icon Gvthandles such cases in the GUI-Comprehension stage

422 Dynamic App GUI-Collection In order to compare thethe mock-up of an app to its implementation Gvt must extractGUI-related meta-data from a running Android app Gvt is able touse Androidrsquos uiautomator framework [2] intended for UI testingto capture xml files and screenshots for a target screen of an apprunning on a physical device or emulator Each uiautomator filecontains information related to the runtime GUI-hierarchy of thetarget app including the following attributes utilized by Gvt (i)The Android component type (eg androidwidgetImageButton)(ii) the location on the screen (iii) the size of the bounding box (iv)text displayed (v) a developer assigned id The hierarchal structureof components is encoded directly in the uiautomator file and thuswe built a parser to extract GUI-hierarchy using this informationdirectly (see Fig 3- 12 )

43 Stage 2 GUI ComprehensionIn order for Gvt to find visual discrepancies between componentsexisting in the mock-up and implementation of an app it must1Similar information regarding mock-up GCs can be parsed from the html or ScalableVector Graphics (svg) format exported by other tools such as Photoshop[1]

determine which components correspond to one another Unfor-tunately the GUI-hierarchies parsed from both the Marketch anduiautomator files tend to differ dramatically due to several factorsmaking tree-based GC matching difficult First since the hierarchyconstructed using the Marketch files is generated using informationfrom the Sketch mock-up of app it is using information derivedfrom designers While designers have tremendous expertise in con-structing visual representations of apps they typically do not takethe time to construct programmatically-oriented groupings of com-ponents Furthermore designers are typically not aware of thecorrect Android component types that should be attributed to dif-ferent objects in a mock-up Second the uiautomator representationof the GUI-hierarchy contains the runtime hierarchal structure ofGCs and correct GC types This tree is typically far more com-plex containing several levels of containers grouping GCs togetherwhich is required for the responsive layouts typical of mobile apps

To overcome this challenge Gvt instead forms two collectionsof leaf-node components from both the mock-up and implementa-tion GUI-hierarchies (Fig 3- 2 ) as this information can be easilyextracted As we reported in Sec 3 the vast majority of DVs af-fects leaf-node components Once the leaf node components havebeen extracted from each hierarchy GVT employs a K-Nearest-Neighbors (KNN) algorithm utilizing a similarity function basedon the location and size of the GCs in order to perform matchingIn this setting an input leaf-node component from the mock-upwould be matched against it closest (eg K=1) neighbor from theimplementation based upon the following similarity function

γ = (|xm minus xr | + |ym minus yr | + |wm minuswr | + |hm minus hr |) (5)

Where γ is a similarity score where smaller values represent closermatches The x yw and h variables correspond to the x amp y lo-cation of the top and left-hand borders of the bounding box andthe height and width of the bounding boxes for the mock-up andimplementation GCs respectively The result is a list of GCs thatshould logically correspond to one another (corresponding GCs)

It is possible that there exist instances of missing or extraneouscomponents between the mock-up and implementation To identifythese cases our KNN algorithm employs a GC-Matching Threshold(MT ) If the similarity score of the nearest neighbor match for agiven input mock-up GC exceeds this threshold it is not matchedwith any component and will be reported as amissing GC violation

ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk

If there are unmatched GCs from the implementation they are laterreported as extraneous GC violations

Also there may be cases where a logical GC in the implemen-tation is represented as small group of mock-up GCs Gvt is ableto handle these cases using the similarity function outlined aboveFor each mock-up GC Gvt checks whether the neighboring GCsin the mockup are closer than the closest corresponding GC inthe implementation If this is the case they are merged with theprocess repeating until a logical GUI-component is represented44 Stage 3 Design Violation DetectionIn the Design Violation Detection stage of the Gvt workflow theapproach uses a combination of computer vision techniques andheuristic checking in order to effectively detect the different cate-gories of DVs derived in our taxonomy presented in Section 3

441 Perceptual Image Differencing In order to determine cor-responding GCs with visual discrepancies Gvt uses a techniquecalled Perceptual Image Differencing (PID) [48] that operates uponthe mock-up and implementation screenshots PID utilizes a modelof the human visual system to compare two images and detectvisual differences and has been used to successfully identify visualdiscrepancies in web applications in previous work [31 32] Weuse this algorithm in conjunction with the GC information derivedin the previous steps of Gvt to achieve accurate violation detec-tion For a full description of the algorithm we refer readers to[48] The PID algorithm uses several adjustable parameters includ-ing F which corresponds to the visual field of view in degrees Lwhich indicates the luminance or brightness of the image and Cwhich adjusts sensitivity to color differences The values used inour implementation are stipulated in Section 45

The output of the PID algorithm is a single difference image (Fig3- 3 ) containing difference pixels which are pixels considered to beperceptually different between the two images After processing thedifference image generated by PID Gvt extracts the implementa-tion bounding box for each corresponding pair of GCs and overlaysthe box on top of the generated difference image It then calculatesthe number of difference pixels contained within the bounding boxwhere higher numbers of difference pixels indicate potential visualdiscrepancies Thus Gvt collects all ldquosuspicious GC pairs with a of difference pixels higher than a Difference Threshold DT This setof suspicious components is then passed to the Violation Manager(Fig 3- 3 ) so that specific instances of DVs can be detected

442 Detecting Layout Violations The first general category ofDVs thatGvt detects are Layout Violations According the taxonomyderived in Sec 3 there are six specific layout DV categories thatrelate to two component properties (i) screen location (ie ltxygtposition) and (ii) size (ie lthwgt of the GC bounding box) Gvt firstchecks for the three types of translationDVs utilizing a heuristic thatmeasures the distance from the top and left-hand edges of matchedcomponents If the difference between the components in either thex ory dimension is greater than a Layout Threshold (LT ) then thesecomponents are reported as a Layout DV Using the LT avoids triviallocation discrepancies within design tolerances being reported asviolations and can be set by a designer or developer using the toolWhen detecting the three types of size DVs in the derived designviolation taxonomyGvt utilizes a heuristic that compares thewidthand height of the bounding boxes of corresponding components If

the width or height of the bounding boxes differ by more than theLT then a layout violation is reported

443 Detecting Text Violations The next general type of DVthat Gvt detects are Text Violations of which there are three spe-cific types (i) Font Color (ii) Font Style and (iii) Incorrect TextContent These detection strategies are only applied to pairs oftext-based components as determined by uiautomator informationTo detect font color violations Gvt extracts cropped images foreach pair of suspicious text components by cropping the mock-upand implementation screenshots according to the componentrsquos re-spective bounding boxes Next Color Quantization (CQ) is appliedto accumulate instances of all unique RGB values expressed in thecomponent-specific images This quantization information is thenused to construct a Color Histogram (CH) (Fig 3- 3 ) Gvt computesthe normalized Euclidean distance between the extracted ColorHistograms for the corresponding GC pairs and if the Histogramsdo not match within a Color Threshold (CT) then a Font-Color DVis reported and the top-3 colors (ie centroids) from each CH arerecorded in theGvt report Likewise if the colors domatch then thePID discrepancy identified earlier is due to the Font-Style changing(provided no existing layout DVs) and thus a Font-Style Violationis reported Finally to detect incorrect text content Gvt utilizesthe textual information preprocessed to remove whitespace andnormalize letter cases and performs a string comparison If thestrings do not match then an Incorrect Text Content DV is reported

444 Detecting Resource Violations Gvt is able to detect thefollowing resource DVs (i) missing component (ii) extraneous com-ponent (iii) image color (iv) incorrect images and (v) componentshape The detection and distinction between Incorrect Image DVsand Image Color DVs requires an analysis that combines two differ-ent computer vision techniques To perform this analysis croppedimages from the mock-up and implementation screenshots accord-ing to corresponding GCs respective bounding boxes are extractedThe goal of this analysis is to determine when the content of image-based GCs differ as opposed to only the colors of the GCs differingTo accomplish this Gvt leverages PID applied to extracted GCimages converted to a binary color space (B-PID) in order to detectdifferences in content and CQ and CH analysis to determine differ-ences in color (Sec 443) To perform the B-PID procedure croppedGC images are converted to a binary color space by extracting pixelintensities and then applying a binary transformation to the inten-sity values (eg converting the images to intensity independentblack amp white) Then PID is run on the color-neutral version ofthese images If the images differ by more than an Image DifferenceThreshold (IDT ) then an Incorrect Image DV (which encompassesthe Component Shape DV ) is reported If the component passes thebinary PID check then Gvt utilizes the same CQ and CH process-ing technique described above to detect image color DVs Missingand extraneous components are detected as described in Sec 43

445 Generating Violation Reports In order to provide develop-ers and designers with effective information regarding the detectedDVs Gvt generates an html report that for each detected viola-tion contains the following (i) a natural language description ofthe design violation(s) (ii) an annotated screenshot of the app im-plementation with the affected GUI-component highlighted (iii)cropped screenshots of the affected GCs from both the design and

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden

implementation screenshots (iv) links to affected lines of applica-tion source code (v) color information extracted from the CH forGCs identified to have color mismatches and (vi) the differenceimage generated by PID The source code links are generated bymatching the ids extracted from the uiautomator information backto their declarations in the layout xml files in the source code (egthose located in the res directory of an apprsquos source code) Weprovide examples of generated reports in our online appendix [35]45 Implementation amp Industrial CollaborationOur implementation of Gvt was developed in Java with a SwingGUI In addition to running the Gvt analysis the tool executableallows for one-click capture of uiautomator files and screenshotsfrom a connected device or emulator Several acceptance tests ofmock-upimplementation screen pairs with pre-existing violationsfrom apps under development within our industrial partner wereused to guide the development of the tool 12 Periodic releases ofbinaries for bothWindows and Mac were made to deploy the tool todesigners and developers within the company The authors of thispaper held regular bi-weekly meetings with members of the designand development teams to plan features and collect feedback

Using the acceptance tests and feedback from our collaboratorswe tuned the various thresholds and parameters of the tool for bestperformance The PID algorithm settings were tuned for sensitivityto capture subtle visual inconsistencies which are then later filteredthrough additional CV techniques F was set to 45 L was set to100cdm2 and C was set to 1 The GC-Matching Threshold (MC)was set to 18th the screen width of a target device the DT fordetermining suspicious GCs was set to 20 The LT was set to 5pixels (based on designer preference) the CT which determinesthe degree to which colors must match for color-based DVs was setto 85 and finally the IDT was set to 20 Gvt allows for a userto change these settings if desired additionally users are capableof defining areas of dynamic content (eg loaded from networkactivity) which should be ignored by the Gvt analysis5 DESIGN OF THE EXPERIMENTSTo evaluate Gvtrsquos performance usefulness and applicability weperform three complimentary studies answering the following RQs

bull RQ1 How well does Gvt perform in terms of detecting andclassifying design violations

bull RQ2 What utility can Gvt provide from the viewpoint ofAndroid developers

bull RQ3What is the industrial applicability of Gvt in terms ofimproving the mobile application development workflow

RQ1 and RQ2 focus on quantitatively measuring the performanceof Gvt and the utility it provides to developers through a controlledempirical and a user study respectively RQ3 reports the results ofa survey and semi-structured interviews with our collaboratorsaimed at investigating the industrial applicability of Gvt51 Study 1 Gvt Effectiveness amp PerformanceThe goal of the first study is to quantitatively measure Gvt in termsof its precision and recall in both detecting and classifying DVs

511 Study Context To carry out a controlled quantitativestudy we manually reverse engineered Sketch mockups for tenscreens for eight of the most popular apps on Google Play To de-rive this set we downloaded the top-10 apps from each category on

the Google-Play store removing the various categories correspond-ing to games (as these have non-standard GUI-components thatGvt does not support) We then randomly sampled one app fromeach of the remaining 33 categories eliminating duplicates (sinceapps can belong to more than one category) We then manuallycollected screenshots and uiautomator files from two screens foreach application using a Nexus 5 attempting to capture the ldquomainrdquoscreen that a user would typically interact with and one secondaryscreen Using the uiautomator files we generated cropped screen-shots of all the leaf nodes components for each screen of the appFrom these we were able generate 10 screens from 8 applicationsthat successfully ran through Gvt without any reported violations

512 Synthetic DV Injection With a set of correct mock-ups cor-responding to implementation screens in an app we needed a suit-able method to introduce DVs into our subjects To this end we con-structed a synthetic DV injection tool that modifies the uiautomator

xml files and corresponding screenshots in order to introduce de-sign violations from our taxonomy presented in Sec 3 The toolis composed of two components (i) an XML Parser that reads andextracts components from the screen then (ii) a Violation Generatorthat randomly selects components and injects synthetic violationsWe implemented injection for the following types of DVsLocation Violation The component is moved either horizontallyvertically or in both directions within the same container Howeverthemaximum distance from the original point is limited by a quarterof the width of the screen size This was based on the severity ofLayout Violations in our study described in Section 3 In order togenerate the image we cropped the component and moved it to thenew location replacing all the original pixels by the most prominentcolor from the surroundings in the original locationSize Violation The component size either increases or decreasesby 20 of the original size For instances where the component sizedecreases we replaced all the pixels by the most prominent colorfrom the surroundings of the original sizeMissing Component Violation This violation removes a leafcomponent from the screen replacing the original pixels by themost prominent color from the surrounding backgroundImage Violation We perturb 40 of the pixels in an image byrandomly generating an RGB value for the pixels affectedImage Color Violation This rule perturbs the color of an imageby shifting the hue of image colors by 30degComponent Color Violation This uses the same process as forImage Color Violations but we change the color by 180degFont Violation This violation randomly selects a font from theset of Arial Comic Sans MS Courier Roboto or Times Roman andapplies it to a TextView componentFont Color Violation changes the text color of a TextView com-ponent We extracted the text color using CH analysis then wechanged the color using same strategy as for Image Color Violations

513 Study Methodology In injecting the synthetic faults wetook several measures to simulate the creation of realistic faultsFirst we delineated 200 different types of design violations accord-ing to the distribution defined in our DV taxonomy in Sec 3 Wethen created a pool of 100 screens by creating random copies ofthe both the uiautomator xml files and screenshots from our initialset of 10 screens We then used the synthetic DV injection tool toseed faults into the pool of 100 screens according to the following

ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk

criteria (i) No screen can contain more than 3 injected DVs (ii)each GC should have a maximum of 1 DV injected and (iii) Eachscreen must have at least 1 injected DV After the DVs were seededeach of the 100 screens and 200 DVs were manually inspected forcorrectness Due to the random nature of the tool a small num-ber of erroneous DVs were excluded and regenerated during thisprocess (eg color perturbed to perceptually similar color) Thebreakdown of injected DVs is shown in Figure 4 and the full datasetwith description is included in our online appendix [35]

Once the final set of screens with injected violations was derivedwe ran Gvt across these subjects and measured four metrics (i)detection precision (DP ) (ii) classification precision (CP ) (iii) recall(R) and (iv) execution time per screen (ET ) We make a distinctionbetween detection and classification in our dataset because it is pos-sible thatGvt is capable of detecting but misclassifying a particularDV (eg an image color DV misclassified as an incorrect image DV )DP CP and R were measured according to the following formulas

DP CP =Tp

Tp + FpR =

Tp

Tp + Fn(6)

where for DP Tp represent injected design violations that weredetected and for CP Tp represents injected violations that wereboth detected and classified correctly In each case Fp correspondto detected DVs that were either not injected or misclassified ForRecallTp represents injected violations that were correctly detectedand Fn represents injected violations that were not detected Tocollect these measures two authors manually examined the reportsfrom Gvt in order to collect the metrics

52 Study 2 Gvt UtilitySince the ultimate goal of an approach like Gvt is to improvethe workflow of developers the goal of this second study is tomeasure the utility (ie benefit) that Gvt provides to developersby investigating two phenomena (i) The accuracy and effort ofdevelopers in detecting and classifying DVs and (ii) the perceivedutility of Gvt reports in helping to identify and resolve DVs

521 Study Context We randomly derived two sets of screensto investigate the two phenomena outlined above First we ran-domly sampled two mutually exclusive sets of 25 and 20 screensrespectively from the 100 used in Study 1 ensuring at least oneinstance of each type of DV was included in the set This resulted inboth sets of screens containing 40 design violations in total The cor-rect mockup screenshot corresponding to each screen sampled fromthe study were also extracted creating pairs of ldquocorrect mockupand ldquoincorrect implementation screenshots 10 participants withat least 5 years of Android development experience were contactedvia email to participate in the survey

522 Study Methodology We created an online survey withfour sections In the first section participants were given back-ground information regarding the definition of DVs and the differ-ent types of DVs derived in our taxonomy In the second sectionparticipants were asked about demographic information such asprogramming experience and education level In the third sectioneach participant was exposed to 5 mock-up implementation screenpairs (displayed side by side on the survey web page) and asked toidentify any observed design violations Descriptions of the DVswere given at the top of this page for reference For each screenpair participants were presented with a dropdown menu to select

80

85

90

95

100

Font

Colo

r(18)

Font

Styl

e(18

)

Imag

e Colo

r(22)

Imag

e(16

)

Miss

ing G

C(26)

Size(1

8)

Text

Conte

nt(1

4)

Trans

lation

(68)

DV Type

Per

cent

age

MetricCPDPR

Figure 4 Study 1 - Detection Precision (DP ) ClassificationPrecision (CP ) and Recall (R)a type for an observed DV and a text field to describe the errorin more detail For each participant one of the 5 mock-up screenswas a control containing no injected violations The 25 screenswere assigned to participants such that each screen was observedby two participants and the order of the screens presented to eachparticipant was randomized to avoid bias To measure the effective-ness of participants in detecting and describing DVs we leveragethe DP CP and R metrics introduced in Study 1 In the fourth sec-tion participants were presented with two screen pairs from thesecond set of 20 sampled from the user study as well as the Gvtreports for these screens Participants were then asked to answer5 user-preferences (UP) and 5 user experience (UX) questions aboutthese reports which are presented in the following section The UPquestions were developed according to the user experience hon-eycomb originally developed by Morville [36] and were posed toparticipants as free form text entry questions We forgo a discussionof the free-form question responses due to space limitations butwe offer full anonymized participant responses in our online ap-pendix [35] We derived the Likert scale-based UX questions usingthe SUS usability scale by Brooke [16]

53 Study 3 Industrial Applicability of GvtThe goal of this final study is determine industrial applicability ofGvt To investigate this we worked with Huawei to collect twosources of information (i) the results of a survey sent to designersand developers who used Gvt in their daily developmentdesignworkflow and (ii) semi-structured interviews with both design anddevelopment managers whose teams have adopted the use of Gvt

531 Study Context ampMethodology We created a survey posingquestions related to the applicability of Gvt to industrial designersand developers These questions are shown in Fig 7 The semi-structured interviews were conducted in Chinese recorded andthen later translated During the interview managers were askedto respond to four questions related to the impact and performanceof the tool in practice We include discussions of the responses inSection 6 and stipulate full questions in our appendix

6 EMPIRICAL RESULTS61 Study 1 Results GVT PerformanceThe results of Study 1 are shown in Figure 4 This figure shows theaverageDP CP and R for each type of seeded violation over the 200seeded faults and the number of faults seeded into each category(following the distributions of our derived taxonomy) are shown onthe x-axis Overall these results are extremely encouraging withthe overall DP achieving 994 the average CP being 984 and

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden

CPDP

R

000 025 050 075 100

Figure 5 Study 2 - Developer CP DP and R

I found these reports unnecessarily complex

I found these reports very cumbersome to read

I think that I would like to use these reports frequently for Identifying Presentation Issues

I thought these reports were very useful for accurately identifying Presentation Errors

These reports are easy to readunderstand

SD D N A SA

Figure 6 Study 2 - UXQuestion Responses SD=Strongly Dis-agree D=Disagree N=Neutral A=Agree SA=Strongly Agree

the average R reaching 965 This illustrates that Gvt is capableof detecting seeded faults designed to emulate both the type anddistribution of DVs encountered in industrial settings While Gvtachieved at least 85 precision for each type of seeded DV it per-formed worse on some types of violations compared to others Forinstance Gvt saw its lowest precision values for the Font-Style andFont-Color violations typically due to the fact that the magnitudeof perturbation for the color or font type was not large enough tosurpass the Color or Image Difference Thresholds (CT amp IDT ) Gvttook 368 mins to process and generate reports for the set of 100screens with injected DVs or 22 sec per screen pair This executioncost was generally acceptable by our industrial collaborators

62 Study 2 Results GVT UtilityThe DP CP and R results representing the Android developersability to correctly detect and classify DVs is shown in Figure 5as box-plots across all 10 participants Here we found CP=DP aswhen a user misclassified violations they also did not detect themAs this figure shows the Android developers generally performedmuch worse compared to Gvt achieving an average CP of underasymp 60 and an average R of asymp 50 The sources of this performanceloss for the study participants compared to Gvt was fourfold (i)participants tended to report minor acceptable differences in fontsacross the examples (despite the instructions clearly stating notto report such violations) (ii) users tended to attribute more thanone DV to a single component specifically for font style and fontcolor violations despite instructions to report only one (iii) userstended to misclassify DVs based on the provided categories (egclassifying a layout DV for a Text GC as an incorrect text DV ) and(iv) participants missed reporting many of the injected DVs lead-ing to the low recall numbers These results indicate that at thevery least developers can struggle to both detect and classify DVsbetween mock-up and implementation screen pairs signaling theneed for an automated system to check for DVs before implementedapps are sent to a UIUX team for auditing This result confirmsthe notion that developers may not be as sensitive to small DVsin the GUI as the designers who created the GUI specificationsFurthermore this finding is notable because as part of the iterativeprocess of resolving design violations designers must communicateto developers DVs and developers must recognize and understandthese DVs in order to properly resolve them This process is oftencomplicated due to ambiguous descriptions of DVs from designers

The GVT allowed for better transfer of the design from mockminusups to the implementation of the app

The GVT has helped you to reduce the time required for verifying design violations

The GVT is able to accurately report existing design violations in productionminusquality applications

The GVT tool helped my team (designimplementation) communicate with other teams (implementationdesign)

regarding GUI design violations

Using the GUIminusVerification tool (GVT) helped to improve the quality of mobile

applications produced by industrial partner

SD D N A SA

Figure 7 Study 3 - Applicability Questions SD=Strongly Dis-agree D=Disagree N=Neutral A=Agree SA=Strongly Agree

to developers or developers disagreeing with designers over the ex-istence or type of a DV In contrast to this fragmented process Gvtprovides clear unambiguous reports that facilitate communicationbetween designers and developers

Figure 6 illustrates the responses to the likert based UX ques-tions and the results are quite encouraging In general participantsfound that the reports from Gvt were easy to read useful for iden-tifying DVs and indicated that they would like to use the reports foridentifying DVs Participants also indicated that the reports werenot unnecessarily complex or difficult to read We asked the partic-ipants about their preferences for the Gvt reports as well askingabout the most and least useful information in the reports Everysingle participant indicated that the highlighted annotations on thescreenshots in the report were the most useful element Whereasmost users tended to dislike the PID output included at the bottomof the report citing this information as difficult to comprehend

63 Study 3 Results Industrial ApplicabilityThe results for the applicability questions asked to 20 designersand developers who use Gvt in their daily activities is shown inFigure 7 A positive outcome for each of these statements correlatesto responses indicating that developers ldquoagreerdquo or ldquostrongly agreerdquoThe results of this study indicate a weak agreement of developersfor these statements indicating that while Gvt is generally appli-cable there are some drawbacks that prevented developers anddesigners from giving the tool unequivocal support We explorethese drawbacks by conducting semi-structured interviews

In conducting the interviews one of the authors asked the ques-tions presented in Figure 7 to 3 managers (2 from UIUX teams and1 from a Front-End development team) When asked whether Gvtcontributed to an increased quality of mobile applications at thecompany all three managers tended to agree that this was the caseFor instance one of the design managers stated ldquoCertainly yes Thetool is the industryrsquos first and the other designer manager addedldquoWhen the page is more complicated the tool is more helpful

When asked about the overall performance and accuracy of thetool in detecting DVs the manager from the implementation teamadmitted that the current detection performance of the tool is goodbut suggested that dynamic detection of some components mayimprove it stating ldquo[DVs ] can be detected pretty well [but the toolis] not very flexible For example a switch component in the designis open but the switch is off in the implementation He suggestedthat properly handling cases such as this would make the toolmore useful from a developers perspective One of the design teammanagers held a similar view stating that ldquoCurrently most errorsare layout errors so tool is accurate Static components are basicallydetected [but] maybe the next extension should focus on dynamic

ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk

components While the current version of the Gvt allows for theexclusion of regions with dynamic components it is clear that bothdesign and development teams would appreciate proper detectionof DVs for dynamic components Additionally two of the managerscommented on the ldquorigidityrdquo of the Gvtrsquos current interface andexplained that a more streamlined UI would help improve its utility

When asked about whether Gvt improved communication be-tween the design and development teams the development teammanager felt that while the tool has not improved communica-tion yet it did have the potential to do so ldquoAt present there is no[improvement] but certainly there is the potential possibility Thedesign managers generally stated that the tool has helped withcommunication particularly in clarifying subtle DVs that may havecaused arguments between teams in the past ldquoIf you consider thetime savings on discussion and arguments between the two teamsthis tool saves us a lot of time Another designer indicated that thetool is helpful at describing DVs to developers who may not be ableto recognize them with the naked eye ldquoWe found that the tool canindeed detect something that the naked eye cannot While there arecertainly further refinements that can be made to Gvt it is clearthat the tool has begun to have a positive impact of the developmentof mobile apps and as the tool evolves within the company shouldallow for continued improvements in quality and time saved

7 LIMITATIONS amp THREATS TO VALIDITYLimitations While we have illustrated thatGvt is applicable in anindustrial setting the tool is not without its limitations Currentlythe tool imposes lightweight restrictions on designers creatingSketch mock-ups chief among these being the requirement thatbounding boxes of components do not overlap Currently Gvt willtry to resolve such cases during the GUI-Comprehension stage usingan Intersection over union (IOU) metricInternal Validity While deriving the taxonomy of DVs mistakesin classification arising from subjectiveness may have introducedunexpected coding Tomitigate this threat we followed a setmethod-ology merged coding results and performed conflict resolutionConstruct Validity In our initial study (Sec 3) a threat to con-struct validity arises in the form of themanner in which coders wereexposed to presentation failures To mitigate this threat designersfrom our industrial partner manually annotated the screen pairsin order to clearly illustrate the affected GCs on the screen In ourevaluation of Gvt threats arise from our method of DV injectionusing the synthetic fault injection tool However we designed thistool to inject faults based upon both the type and distribution offaults from our DV taxonomy to mitigate this threatExternal Validity In our initial study related to the DV taxonomywe utilized a dataset from a single (albeit large) company withexamples across several different applications and screens There isthe potential that this may not generalize to other industrial mobileapplication development environments and platforms or mobile appdevelopment in general However given the relatively consistentdesign paradigms of mobile apps we expect the categories andthe sub-categories within the taxonomy to hold although it ispossible that the distribution across these categories may varyacross application development for different domains In Study 3we surveyed employees at a single (though large) company andfindings may differ in similar studies at other companies

8 RELATEDWORKWeb Presentation Failures The work most closely related to ourapproach are approaches that aim at detecting classifying and fixingpresentation failures in web applications [30ndash32 42] In comparisonto these approaches Gvt also performs detection and localizationof presentation failures but is the first to do so for mobile appsIn addition to the engineering challenges associated with buildingan approach to detect presentation failures in the mobile domain(eg collection and processing of GUI-related data) Gvt is the firstapproach to leverage metadata from software mock-up artifacts(eg Marketch) to perform GC matching based upon the spatialinformation collected from both mock-ups and dynamic applicationscreens allowing for precise detection of the different types of DVsdelineated in our industrial DV taxonomy Gvt is also the first toapply the processes of CQ CH analysis and B-PID toward detectingdifferences in the content and color of icons and images displayedin mobile apps Gvt also explicitly identifies and reports differentfaulty properties (such as errors in component location or text)Cross Browser Testing Approaches for XBT (or cross browsertesting) by Roy Choudhry et al [18 42 43] examine and automati-cally report differences in web pages rendered in multiple browsersThese approaches are currently not directly applicable to mock-updriven development for mobile appsVisualGUI Testing A concept known as Visual GUI Testing (VGT)aims to test certain visual aspects of a software applicationrsquos GUIas well as the underlying functional properties To accomplish thisvisual GUI testing usually executes actions on a target applicationsin order to exercise app functionality [13 14 23 38] In contrastto these approaches Gvt is designed to apply to mobile-specificDVs is tailored for the mock-up driven development practice andis aimed only at verifying visual properties of a mobile apprsquos GUIOther Approaches There are other approaches and techniquesthat related to identifying problems or differences with GUIs ofmobile apps Xie et al introduced GUIDE [47] a tool for GUI differ-encing between successive releases of GUIs for an app by matchingcomponents between GUI-hierarchies Gvt utilizes a matching pro-cedure for leaf node components as direct tree comparisons are notpossible in the context of mock-up driven development There hasalso been both commercial and academic work related to graphi-cal software built specifically for creating high-fidelity mobile appmock-ups or mockups that encode information for automated cre-ation of code for a target platform [4 8 9 34] However such toolstend to either impose too many restrictions on designers or do notallow for direct creation of code thus DVs still persist in practice9 CONCLUSION amp FUTUREWORKIn this paper we have formalized the problem of detecting designviolations in mobile apps and derived a taxonomy of design viola-tions based on a robust industrial dataset We presented Gvt anapproach for automatically detecting classifying and reporting de-sign violations in mobile apps and conducted a wide ranging studythat measured performance utility and industrial applicability ofthis tool Our results indicate that Gvt is effective in practice offersutility for developers and is applicable in industrial contexts

ACKNOWLEDGMENTSThe authors would like to thank Kebing Xie Roozbeh Farahbodand the developers and designers at Huawei for their support

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden

REFERENCES[1] Adobe photoshop httpwwwphotoshopcom[2] Android uiautomator httpdeveloperandroidcomtoolshelpuiautomator

indexhtml[3] Apple app store httpswwwapplecomiosapp-store[4] Fuild-ui httpswwwfluiduicom[5] Google play store httpsplaygooglecomstorehl=en[6] The marketch plugin for sketch httpsgithubcomtudou527marketch[7] Mobile apps What consumers really need and want httpsinfodynatracecom

rscompuwareimagesMobileAppSurveyReportpdf[8] Mockupio httpsmockupioabout[9] Protoio httpsprotoio[10] The sketch design tool httpswwwsketchappcom[11] Sketch extensions httpswwwsketchappcomextensions[12] Why your apprsquos ux is more important than you think httpwwwcodemagcom

Article1401041[13] E Aleacutegroth and R Feldt On the long-term use of visual gui testing in industrial

practice a case study Empirical Software Engineering 22(6)2937ndash2971 Dec 2017[14] E Alegroth Z Gao R Oliveira and A Memon Conceptualization and evaluation

of component-based testing unified with visual gui testing An empirical studyIn 2015 IEEE 8th International Conference on Software Testing Verification andValidation (ICST) pages 1ndash10 April 2015

[15] G Bavota M Linares-Vaacutesquez C Bernal-Caacuterdenas M Di Penta R Oliveto andD Poshyvanyk The impact of api change- and fault-proneness on the user ratingsof android apps Software Engineering IEEE Transactions on 41(4)384ndash407 April2015

[16] J Brooke SUS A quick and dirty usability scale In P W Jordan B WeerdmeesterA Thomas and I L Mclelland editors Usability evaluation in industry Taylorand Francis London 1996

[17] K Charmaz Constructing Grounded Theory SAGE Publications Inc 2006[18] S R Choudhary M R Prasad and A Orso Crosscheck Combining crawling and

differencing to better detect cross-browser incompatibilities in web applicationsIn Proceedings of the 2012 IEEE Fifth International Conference on Software TestingVerification and Validation ICST rsquo12 pages 171ndash180 Washington DC USA 2012IEEE Computer Society

[19] A Ciurumelea A SchaufelbAtildeČAcircijhl S Panichella and H C Gall Analyzingreviews and code of mobile apps for better release planning In 2017 IEEE 24th In-ternational Conference on Software Analysis Evolution and Reengineering (SANER)pages 91ndash102 Feb 2017

[20] A Di Sorbo S Panichella C V Alexandru J Shimagaki C A Visaggio G Can-fora and H C Gall What would users change in my app summarizing appreviews for recommending software changes In Proceedings of the 2016 24thACM SIGSOFT International Symposium on Foundations of Software EngineeringFSE 2016 pages 499ndash510 New York NY USA 2016 ACM

[21] K Holl and F Elberzhager A mobile-specific failure classification and its usageto focus quality assurance In 2014 40th EUROMICRO Conference on SoftwareEngineering and Advanced Applications pages 385ndash388 Aug 2014

[22] G Hu X Yuan Y Tang and J Yang Efficiently effectively detecting mobileapp bugs with appdoctor In Proceedings of the Ninth European Conference onComputer Systems EuroSys rsquo14 pages 181ndash1815 New York NY USA 2014ACM

[23] A Issa J Sillito and V Garousi Visual testing of graphical user interfaces Anexploratory study towards systematic definitions and approaches In 2012 14thIEEE International Symposium on Web Systems Evolution (WSE) pages 11ndash15 Sept2012

[24] N Jones Seven best practices for optimizing mobile testing efforts TechnicalReport G00248240 Gartner

[25] K Kuusinen and T Mikkonen Designing user experience for mobile apps Long-term product owner perspective In 2013 20th Asia-Pacific Software EngineeringConference volume 1 of APSECrsquo13 pages 535ndash540 Dec 2013

[26] V Lelli A Blouin and B Baudry Classifying and qualifying gui defects In 2015IEEE 8th International Conference on Software Testing Verification and Validation(ICST) pages 1ndash10 April 2015

[27] M Linares-Vaacutesquez G Bavota C Bernal-Caacuterdenas M Di Penta R Oliveto andD Poshyvanyk Api change and fault proneness A threat to the success ofandroid apps In Proceedings of the 2013 9th Joint Meeting on Foundations ofSoftware Engineering ESECFSErsquo13 pages 477ndash487 New York NY USA 2013ACM

[28] M Linares-Vaacutesquez G Bavota M D Penta R Oliveto and D Poshyvanyk Howdo API changes trigger Stack Overflow discussions a study on the android SDK

In Proceedings of the 22nd International Conference on Program ComprehensionICPCrsquo14 pages 83ndash94 2014

[29] M Linares-Vaacutesquez G Bavota M Tufano K Moran M Di Penta C VendomeC Bernal-Caacuterdenas and D Poshyvanyk Enabling mutation testing for androidapps In Proceedings of the 2017 11th Joint Meeting on Foundations of SoftwareEngineering ESECFSE 2017 pages 233ndash244 New York NY USA 2017 ACM

[30] S Mahajan A Alameer P McMinn and W G Halfond Automated repair oflayout cross browser issues using search-based techniques In InternationalConference on Software Testing and Analysis ISSTArsquo17 2017

[31] S Mahajan and W G J Halfond Detection and localization of html presentationfailures using computer vision-based techniques In Proceedings of the 8th IEEEInternational Conference on Software Testing Verification and Validation ICSTrsquo15April 2015

[32] S Mahajan B Li P Behnamghader and W G Halfond Using visual symptomsfor debugging presentation failures in web applications In Proceeding of the9th IEEE International Conference on Software Testing Verification and Validation(ICST) ICSTrsquo16 April 2016

[33] T McDonnell B Ray andM Kim An empirical study of api stability and adoptionin the android ecosystem In Proceedings of the 2013 International Conference onSoftware Maintenance ICSMrsquo13 pages 70ndash79 2013

[34] J Meskens K Luyten and K Coninx Plug-and-design Embracing mobiledevices as part of the design environment In Proceedings of the 1st ACM SIGCHISymposium on Engineering Interactive Computing Systems EICS rsquo09 pages 149ndash154 New York NY USA 2009 ACM

[35] K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk Gvt onlineappendix httpwwwandroid-dev-toolscomgvt

[36] P Morville User experience design httpsemanticstudioscomuser_experience_design

[37] B Myers Challenges of hci design and implementation interactions 1(1)73ndash83Jan 1994

[38] B N Nguyen B Robbins I Banerjee and A Memon Guitar An innovativetool for automated testing of gui-driven software Automated Software Engg21(1)65ndash105 Mar 2014

[39] T A Nguyen and C Csallner Reverse engineering mobile application userinterfaces with REMAUI In Proceedings of the 2015 30th IEEEACM Interna-tional Conference on Automated Software Engineering ASErsquo15 pages 248ndash259Washington DC USA 2015 IEEE Computer Society

[40] F PalombaM Linares-Vaacutesquez G Bavota R Oliveto M D Penta D Poshyvanykand A D Lucia User reviews matter tracking crowdsourced reviews to supportevolution of successful apps In 2015 IEEE International Conference on SoftwareMaintenance and Evolution (ICSME) pages 291ndash300 Sept 2015

[41] F Palomba P Salza A Ciurumelea S Panichella H Gall F Ferrucci andA De Lucia Recommending and localizing change requests for mobile appsbased on user reviews In Proceedings of the 39th International Conference onSoftware Engineering ICSE rsquo17 pages 106ndash117 Piscataway NJ USA 2017 IEEEPress

[42] S Roy Choudhary M R Prasad and A Orso X-pert Accurate identification ofcross-browser issues in web applications In Proceedings of the 2013 InternationalConference on Software Engineering ICSE rsquo13 pages 702ndash711 Piscataway NJUSA 2013 IEEE Press

[43] S Roy Choudhary H Versee and A Orso Webdiff Automated identification ofcross-browser issues in web applications In Proceedings of the 2010 IEEE Interna-tional Conference on Software Maintenance ICSM rsquo10 pages 1ndash10 WashingtonDC USA 2010 IEEE Computer Society

[44] T Silva da Silva A Martin F Maurer and M Silveira User-centered design andagile methods A systematic review In Proceedings of the 2011 Agile ConferenceAGILE rsquo11 pages 77ndash86 Washington DC USA 2011 IEEE Computer Society

[45] K-J Stol P Ralph and B Fitzgerald Grounded theory in software engineeringresearch A critical review and guidelines In Proceedings of the 38th InternationalConference on Software Engineering ICSE rsquo16 pages 120ndash131 New York NY USA2016 ACM

[46] A B Tucker Computer Science Handbook Second Edition Chapman amp HallCRC2004

[47] Q Xie M Grechanik C Fu and C Cumby Guide A gui differentiator In 2009IEEE International Conference on Software Maintenance ICSMrsquo09

[48] H Yee S Pattanaik and D P Greenberg Spatiotemporal sensitivity and visualattention for efficient rendering of dynamic environments ACM Trans Graph20(1)39ndash65 Jan 2001

[49] C Zeidler C Lutteroth W Stuerzlinger and G Weber Evaluating Direct Manip-ulation Operations for Constraint-Based Layout pages 513ndash529 Springer BerlinHeidelberg Berlin Heidelberg 2013

ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk

If there are unmatched GCs from the implementation they are laterreported as extraneous GC violations

Also there may be cases where a logical GC in the implemen-tation is represented as small group of mock-up GCs Gvt is ableto handle these cases using the similarity function outlined aboveFor each mock-up GC Gvt checks whether the neighboring GCsin the mockup are closer than the closest corresponding GC inthe implementation If this is the case they are merged with theprocess repeating until a logical GUI-component is represented44 Stage 3 Design Violation DetectionIn the Design Violation Detection stage of the Gvt workflow theapproach uses a combination of computer vision techniques andheuristic checking in order to effectively detect the different cate-gories of DVs derived in our taxonomy presented in Section 3

441 Perceptual Image Differencing In order to determine cor-responding GCs with visual discrepancies Gvt uses a techniquecalled Perceptual Image Differencing (PID) [48] that operates uponthe mock-up and implementation screenshots PID utilizes a modelof the human visual system to compare two images and detectvisual differences and has been used to successfully identify visualdiscrepancies in web applications in previous work [31 32] Weuse this algorithm in conjunction with the GC information derivedin the previous steps of Gvt to achieve accurate violation detec-tion For a full description of the algorithm we refer readers to[48] The PID algorithm uses several adjustable parameters includ-ing F which corresponds to the visual field of view in degrees Lwhich indicates the luminance or brightness of the image and Cwhich adjusts sensitivity to color differences The values used inour implementation are stipulated in Section 45

The output of the PID algorithm is a single difference image (Fig3- 3 ) containing difference pixels which are pixels considered to beperceptually different between the two images After processing thedifference image generated by PID Gvt extracts the implementa-tion bounding box for each corresponding pair of GCs and overlaysthe box on top of the generated difference image It then calculatesthe number of difference pixels contained within the bounding boxwhere higher numbers of difference pixels indicate potential visualdiscrepancies Thus Gvt collects all ldquosuspicious GC pairs with a of difference pixels higher than a Difference Threshold DT This setof suspicious components is then passed to the Violation Manager(Fig 3- 3 ) so that specific instances of DVs can be detected

442 Detecting Layout Violations The first general category ofDVs thatGvt detects are Layout Violations According the taxonomyderived in Sec 3 there are six specific layout DV categories thatrelate to two component properties (i) screen location (ie ltxygtposition) and (ii) size (ie lthwgt of the GC bounding box) Gvt firstchecks for the three types of translationDVs utilizing a heuristic thatmeasures the distance from the top and left-hand edges of matchedcomponents If the difference between the components in either thex ory dimension is greater than a Layout Threshold (LT ) then thesecomponents are reported as a Layout DV Using the LT avoids triviallocation discrepancies within design tolerances being reported asviolations and can be set by a designer or developer using the toolWhen detecting the three types of size DVs in the derived designviolation taxonomyGvt utilizes a heuristic that compares thewidthand height of the bounding boxes of corresponding components If

the width or height of the bounding boxes differ by more than theLT then a layout violation is reported

443 Detecting Text Violations The next general type of DVthat Gvt detects are Text Violations of which there are three spe-cific types (i) Font Color (ii) Font Style and (iii) Incorrect TextContent These detection strategies are only applied to pairs oftext-based components as determined by uiautomator informationTo detect font color violations Gvt extracts cropped images foreach pair of suspicious text components by cropping the mock-upand implementation screenshots according to the componentrsquos re-spective bounding boxes Next Color Quantization (CQ) is appliedto accumulate instances of all unique RGB values expressed in thecomponent-specific images This quantization information is thenused to construct a Color Histogram (CH) (Fig 3- 3 ) Gvt computesthe normalized Euclidean distance between the extracted ColorHistograms for the corresponding GC pairs and if the Histogramsdo not match within a Color Threshold (CT) then a Font-Color DVis reported and the top-3 colors (ie centroids) from each CH arerecorded in theGvt report Likewise if the colors domatch then thePID discrepancy identified earlier is due to the Font-Style changing(provided no existing layout DVs) and thus a Font-Style Violationis reported Finally to detect incorrect text content Gvt utilizesthe textual information preprocessed to remove whitespace andnormalize letter cases and performs a string comparison If thestrings do not match then an Incorrect Text Content DV is reported

444 Detecting Resource Violations Gvt is able to detect thefollowing resource DVs (i) missing component (ii) extraneous com-ponent (iii) image color (iv) incorrect images and (v) componentshape The detection and distinction between Incorrect Image DVsand Image Color DVs requires an analysis that combines two differ-ent computer vision techniques To perform this analysis croppedimages from the mock-up and implementation screenshots accord-ing to corresponding GCs respective bounding boxes are extractedThe goal of this analysis is to determine when the content of image-based GCs differ as opposed to only the colors of the GCs differingTo accomplish this Gvt leverages PID applied to extracted GCimages converted to a binary color space (B-PID) in order to detectdifferences in content and CQ and CH analysis to determine differ-ences in color (Sec 443) To perform the B-PID procedure croppedGC images are converted to a binary color space by extracting pixelintensities and then applying a binary transformation to the inten-sity values (eg converting the images to intensity independentblack amp white) Then PID is run on the color-neutral version ofthese images If the images differ by more than an Image DifferenceThreshold (IDT ) then an Incorrect Image DV (which encompassesthe Component Shape DV ) is reported If the component passes thebinary PID check then Gvt utilizes the same CQ and CH process-ing technique described above to detect image color DVs Missingand extraneous components are detected as described in Sec 43

445 Generating Violation Reports In order to provide develop-ers and designers with effective information regarding the detectedDVs Gvt generates an html report that for each detected viola-tion contains the following (i) a natural language description ofthe design violation(s) (ii) an annotated screenshot of the app im-plementation with the affected GUI-component highlighted (iii)cropped screenshots of the affected GCs from both the design and

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden

implementation screenshots (iv) links to affected lines of applica-tion source code (v) color information extracted from the CH forGCs identified to have color mismatches and (vi) the differenceimage generated by PID The source code links are generated bymatching the ids extracted from the uiautomator information backto their declarations in the layout xml files in the source code (egthose located in the res directory of an apprsquos source code) Weprovide examples of generated reports in our online appendix [35]45 Implementation amp Industrial CollaborationOur implementation of Gvt was developed in Java with a SwingGUI In addition to running the Gvt analysis the tool executableallows for one-click capture of uiautomator files and screenshotsfrom a connected device or emulator Several acceptance tests ofmock-upimplementation screen pairs with pre-existing violationsfrom apps under development within our industrial partner wereused to guide the development of the tool 12 Periodic releases ofbinaries for bothWindows and Mac were made to deploy the tool todesigners and developers within the company The authors of thispaper held regular bi-weekly meetings with members of the designand development teams to plan features and collect feedback

Using the acceptance tests and feedback from our collaboratorswe tuned the various thresholds and parameters of the tool for bestperformance The PID algorithm settings were tuned for sensitivityto capture subtle visual inconsistencies which are then later filteredthrough additional CV techniques F was set to 45 L was set to100cdm2 and C was set to 1 The GC-Matching Threshold (MC)was set to 18th the screen width of a target device the DT fordetermining suspicious GCs was set to 20 The LT was set to 5pixels (based on designer preference) the CT which determinesthe degree to which colors must match for color-based DVs was setto 85 and finally the IDT was set to 20 Gvt allows for a userto change these settings if desired additionally users are capableof defining areas of dynamic content (eg loaded from networkactivity) which should be ignored by the Gvt analysis5 DESIGN OF THE EXPERIMENTSTo evaluate Gvtrsquos performance usefulness and applicability weperform three complimentary studies answering the following RQs

bull RQ1 How well does Gvt perform in terms of detecting andclassifying design violations

bull RQ2 What utility can Gvt provide from the viewpoint ofAndroid developers

bull RQ3What is the industrial applicability of Gvt in terms ofimproving the mobile application development workflow

RQ1 and RQ2 focus on quantitatively measuring the performanceof Gvt and the utility it provides to developers through a controlledempirical and a user study respectively RQ3 reports the results ofa survey and semi-structured interviews with our collaboratorsaimed at investigating the industrial applicability of Gvt51 Study 1 Gvt Effectiveness amp PerformanceThe goal of the first study is to quantitatively measure Gvt in termsof its precision and recall in both detecting and classifying DVs

511 Study Context To carry out a controlled quantitativestudy we manually reverse engineered Sketch mockups for tenscreens for eight of the most popular apps on Google Play To de-rive this set we downloaded the top-10 apps from each category on

the Google-Play store removing the various categories correspond-ing to games (as these have non-standard GUI-components thatGvt does not support) We then randomly sampled one app fromeach of the remaining 33 categories eliminating duplicates (sinceapps can belong to more than one category) We then manuallycollected screenshots and uiautomator files from two screens foreach application using a Nexus 5 attempting to capture the ldquomainrdquoscreen that a user would typically interact with and one secondaryscreen Using the uiautomator files we generated cropped screen-shots of all the leaf nodes components for each screen of the appFrom these we were able generate 10 screens from 8 applicationsthat successfully ran through Gvt without any reported violations

512 Synthetic DV Injection With a set of correct mock-ups cor-responding to implementation screens in an app we needed a suit-able method to introduce DVs into our subjects To this end we con-structed a synthetic DV injection tool that modifies the uiautomator

xml files and corresponding screenshots in order to introduce de-sign violations from our taxonomy presented in Sec 3 The toolis composed of two components (i) an XML Parser that reads andextracts components from the screen then (ii) a Violation Generatorthat randomly selects components and injects synthetic violationsWe implemented injection for the following types of DVsLocation Violation The component is moved either horizontallyvertically or in both directions within the same container Howeverthemaximum distance from the original point is limited by a quarterof the width of the screen size This was based on the severity ofLayout Violations in our study described in Section 3 In order togenerate the image we cropped the component and moved it to thenew location replacing all the original pixels by the most prominentcolor from the surroundings in the original locationSize Violation The component size either increases or decreasesby 20 of the original size For instances where the component sizedecreases we replaced all the pixels by the most prominent colorfrom the surroundings of the original sizeMissing Component Violation This violation removes a leafcomponent from the screen replacing the original pixels by themost prominent color from the surrounding backgroundImage Violation We perturb 40 of the pixels in an image byrandomly generating an RGB value for the pixels affectedImage Color Violation This rule perturbs the color of an imageby shifting the hue of image colors by 30degComponent Color Violation This uses the same process as forImage Color Violations but we change the color by 180degFont Violation This violation randomly selects a font from theset of Arial Comic Sans MS Courier Roboto or Times Roman andapplies it to a TextView componentFont Color Violation changes the text color of a TextView com-ponent We extracted the text color using CH analysis then wechanged the color using same strategy as for Image Color Violations

513 Study Methodology In injecting the synthetic faults wetook several measures to simulate the creation of realistic faultsFirst we delineated 200 different types of design violations accord-ing to the distribution defined in our DV taxonomy in Sec 3 Wethen created a pool of 100 screens by creating random copies ofthe both the uiautomator xml files and screenshots from our initialset of 10 screens We then used the synthetic DV injection tool toseed faults into the pool of 100 screens according to the following

ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk

criteria (i) No screen can contain more than 3 injected DVs (ii)each GC should have a maximum of 1 DV injected and (iii) Eachscreen must have at least 1 injected DV After the DVs were seededeach of the 100 screens and 200 DVs were manually inspected forcorrectness Due to the random nature of the tool a small num-ber of erroneous DVs were excluded and regenerated during thisprocess (eg color perturbed to perceptually similar color) Thebreakdown of injected DVs is shown in Figure 4 and the full datasetwith description is included in our online appendix [35]

Once the final set of screens with injected violations was derivedwe ran Gvt across these subjects and measured four metrics (i)detection precision (DP ) (ii) classification precision (CP ) (iii) recall(R) and (iv) execution time per screen (ET ) We make a distinctionbetween detection and classification in our dataset because it is pos-sible thatGvt is capable of detecting but misclassifying a particularDV (eg an image color DV misclassified as an incorrect image DV )DP CP and R were measured according to the following formulas

DP CP =Tp

Tp + FpR =

Tp

Tp + Fn(6)

where for DP Tp represent injected design violations that weredetected and for CP Tp represents injected violations that wereboth detected and classified correctly In each case Fp correspondto detected DVs that were either not injected or misclassified ForRecallTp represents injected violations that were correctly detectedand Fn represents injected violations that were not detected Tocollect these measures two authors manually examined the reportsfrom Gvt in order to collect the metrics

52 Study 2 Gvt UtilitySince the ultimate goal of an approach like Gvt is to improvethe workflow of developers the goal of this second study is tomeasure the utility (ie benefit) that Gvt provides to developersby investigating two phenomena (i) The accuracy and effort ofdevelopers in detecting and classifying DVs and (ii) the perceivedutility of Gvt reports in helping to identify and resolve DVs

521 Study Context We randomly derived two sets of screensto investigate the two phenomena outlined above First we ran-domly sampled two mutually exclusive sets of 25 and 20 screensrespectively from the 100 used in Study 1 ensuring at least oneinstance of each type of DV was included in the set This resulted inboth sets of screens containing 40 design violations in total The cor-rect mockup screenshot corresponding to each screen sampled fromthe study were also extracted creating pairs of ldquocorrect mockupand ldquoincorrect implementation screenshots 10 participants withat least 5 years of Android development experience were contactedvia email to participate in the survey

522 Study Methodology We created an online survey withfour sections In the first section participants were given back-ground information regarding the definition of DVs and the differ-ent types of DVs derived in our taxonomy In the second sectionparticipants were asked about demographic information such asprogramming experience and education level In the third sectioneach participant was exposed to 5 mock-up implementation screenpairs (displayed side by side on the survey web page) and asked toidentify any observed design violations Descriptions of the DVswere given at the top of this page for reference For each screenpair participants were presented with a dropdown menu to select

80

85

90

95

100

Font

Colo

r(18)

Font

Styl

e(18

)

Imag

e Colo

r(22)

Imag

e(16

)

Miss

ing G

C(26)

Size(1

8)

Text

Conte

nt(1

4)

Trans

lation

(68)

DV Type

Per

cent

age

MetricCPDPR

Figure 4 Study 1 - Detection Precision (DP ) ClassificationPrecision (CP ) and Recall (R)a type for an observed DV and a text field to describe the errorin more detail For each participant one of the 5 mock-up screenswas a control containing no injected violations The 25 screenswere assigned to participants such that each screen was observedby two participants and the order of the screens presented to eachparticipant was randomized to avoid bias To measure the effective-ness of participants in detecting and describing DVs we leveragethe DP CP and R metrics introduced in Study 1 In the fourth sec-tion participants were presented with two screen pairs from thesecond set of 20 sampled from the user study as well as the Gvtreports for these screens Participants were then asked to answer5 user-preferences (UP) and 5 user experience (UX) questions aboutthese reports which are presented in the following section The UPquestions were developed according to the user experience hon-eycomb originally developed by Morville [36] and were posed toparticipants as free form text entry questions We forgo a discussionof the free-form question responses due to space limitations butwe offer full anonymized participant responses in our online ap-pendix [35] We derived the Likert scale-based UX questions usingthe SUS usability scale by Brooke [16]

53 Study 3 Industrial Applicability of GvtThe goal of this final study is determine industrial applicability ofGvt To investigate this we worked with Huawei to collect twosources of information (i) the results of a survey sent to designersand developers who used Gvt in their daily developmentdesignworkflow and (ii) semi-structured interviews with both design anddevelopment managers whose teams have adopted the use of Gvt

531 Study Context ampMethodology We created a survey posingquestions related to the applicability of Gvt to industrial designersand developers These questions are shown in Fig 7 The semi-structured interviews were conducted in Chinese recorded andthen later translated During the interview managers were askedto respond to four questions related to the impact and performanceof the tool in practice We include discussions of the responses inSection 6 and stipulate full questions in our appendix

6 EMPIRICAL RESULTS61 Study 1 Results GVT PerformanceThe results of Study 1 are shown in Figure 4 This figure shows theaverageDP CP and R for each type of seeded violation over the 200seeded faults and the number of faults seeded into each category(following the distributions of our derived taxonomy) are shown onthe x-axis Overall these results are extremely encouraging withthe overall DP achieving 994 the average CP being 984 and

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden

CPDP

R

000 025 050 075 100

Figure 5 Study 2 - Developer CP DP and R

I found these reports unnecessarily complex

I found these reports very cumbersome to read

I think that I would like to use these reports frequently for Identifying Presentation Issues

I thought these reports were very useful for accurately identifying Presentation Errors

These reports are easy to readunderstand

SD D N A SA

Figure 6 Study 2 - UXQuestion Responses SD=Strongly Dis-agree D=Disagree N=Neutral A=Agree SA=Strongly Agree

the average R reaching 965 This illustrates that Gvt is capableof detecting seeded faults designed to emulate both the type anddistribution of DVs encountered in industrial settings While Gvtachieved at least 85 precision for each type of seeded DV it per-formed worse on some types of violations compared to others Forinstance Gvt saw its lowest precision values for the Font-Style andFont-Color violations typically due to the fact that the magnitudeof perturbation for the color or font type was not large enough tosurpass the Color or Image Difference Thresholds (CT amp IDT ) Gvttook 368 mins to process and generate reports for the set of 100screens with injected DVs or 22 sec per screen pair This executioncost was generally acceptable by our industrial collaborators

62 Study 2 Results GVT UtilityThe DP CP and R results representing the Android developersability to correctly detect and classify DVs is shown in Figure 5as box-plots across all 10 participants Here we found CP=DP aswhen a user misclassified violations they also did not detect themAs this figure shows the Android developers generally performedmuch worse compared to Gvt achieving an average CP of underasymp 60 and an average R of asymp 50 The sources of this performanceloss for the study participants compared to Gvt was fourfold (i)participants tended to report minor acceptable differences in fontsacross the examples (despite the instructions clearly stating notto report such violations) (ii) users tended to attribute more thanone DV to a single component specifically for font style and fontcolor violations despite instructions to report only one (iii) userstended to misclassify DVs based on the provided categories (egclassifying a layout DV for a Text GC as an incorrect text DV ) and(iv) participants missed reporting many of the injected DVs lead-ing to the low recall numbers These results indicate that at thevery least developers can struggle to both detect and classify DVsbetween mock-up and implementation screen pairs signaling theneed for an automated system to check for DVs before implementedapps are sent to a UIUX team for auditing This result confirmsthe notion that developers may not be as sensitive to small DVsin the GUI as the designers who created the GUI specificationsFurthermore this finding is notable because as part of the iterativeprocess of resolving design violations designers must communicateto developers DVs and developers must recognize and understandthese DVs in order to properly resolve them This process is oftencomplicated due to ambiguous descriptions of DVs from designers

The GVT allowed for better transfer of the design from mockminusups to the implementation of the app

The GVT has helped you to reduce the time required for verifying design violations

The GVT is able to accurately report existing design violations in productionminusquality applications

The GVT tool helped my team (designimplementation) communicate with other teams (implementationdesign)

regarding GUI design violations

Using the GUIminusVerification tool (GVT) helped to improve the quality of mobile

applications produced by industrial partner

SD D N A SA

Figure 7 Study 3 - Applicability Questions SD=Strongly Dis-agree D=Disagree N=Neutral A=Agree SA=Strongly Agree

to developers or developers disagreeing with designers over the ex-istence or type of a DV In contrast to this fragmented process Gvtprovides clear unambiguous reports that facilitate communicationbetween designers and developers

Figure 6 illustrates the responses to the likert based UX ques-tions and the results are quite encouraging In general participantsfound that the reports from Gvt were easy to read useful for iden-tifying DVs and indicated that they would like to use the reports foridentifying DVs Participants also indicated that the reports werenot unnecessarily complex or difficult to read We asked the partic-ipants about their preferences for the Gvt reports as well askingabout the most and least useful information in the reports Everysingle participant indicated that the highlighted annotations on thescreenshots in the report were the most useful element Whereasmost users tended to dislike the PID output included at the bottomof the report citing this information as difficult to comprehend

63 Study 3 Results Industrial ApplicabilityThe results for the applicability questions asked to 20 designersand developers who use Gvt in their daily activities is shown inFigure 7 A positive outcome for each of these statements correlatesto responses indicating that developers ldquoagreerdquo or ldquostrongly agreerdquoThe results of this study indicate a weak agreement of developersfor these statements indicating that while Gvt is generally appli-cable there are some drawbacks that prevented developers anddesigners from giving the tool unequivocal support We explorethese drawbacks by conducting semi-structured interviews

In conducting the interviews one of the authors asked the ques-tions presented in Figure 7 to 3 managers (2 from UIUX teams and1 from a Front-End development team) When asked whether Gvtcontributed to an increased quality of mobile applications at thecompany all three managers tended to agree that this was the caseFor instance one of the design managers stated ldquoCertainly yes Thetool is the industryrsquos first and the other designer manager addedldquoWhen the page is more complicated the tool is more helpful

When asked about the overall performance and accuracy of thetool in detecting DVs the manager from the implementation teamadmitted that the current detection performance of the tool is goodbut suggested that dynamic detection of some components mayimprove it stating ldquo[DVs ] can be detected pretty well [but the toolis] not very flexible For example a switch component in the designis open but the switch is off in the implementation He suggestedthat properly handling cases such as this would make the toolmore useful from a developers perspective One of the design teammanagers held a similar view stating that ldquoCurrently most errorsare layout errors so tool is accurate Static components are basicallydetected [but] maybe the next extension should focus on dynamic

ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk

components While the current version of the Gvt allows for theexclusion of regions with dynamic components it is clear that bothdesign and development teams would appreciate proper detectionof DVs for dynamic components Additionally two of the managerscommented on the ldquorigidityrdquo of the Gvtrsquos current interface andexplained that a more streamlined UI would help improve its utility

When asked about whether Gvt improved communication be-tween the design and development teams the development teammanager felt that while the tool has not improved communica-tion yet it did have the potential to do so ldquoAt present there is no[improvement] but certainly there is the potential possibility Thedesign managers generally stated that the tool has helped withcommunication particularly in clarifying subtle DVs that may havecaused arguments between teams in the past ldquoIf you consider thetime savings on discussion and arguments between the two teamsthis tool saves us a lot of time Another designer indicated that thetool is helpful at describing DVs to developers who may not be ableto recognize them with the naked eye ldquoWe found that the tool canindeed detect something that the naked eye cannot While there arecertainly further refinements that can be made to Gvt it is clearthat the tool has begun to have a positive impact of the developmentof mobile apps and as the tool evolves within the company shouldallow for continued improvements in quality and time saved

7 LIMITATIONS amp THREATS TO VALIDITYLimitations While we have illustrated thatGvt is applicable in anindustrial setting the tool is not without its limitations Currentlythe tool imposes lightweight restrictions on designers creatingSketch mock-ups chief among these being the requirement thatbounding boxes of components do not overlap Currently Gvt willtry to resolve such cases during the GUI-Comprehension stage usingan Intersection over union (IOU) metricInternal Validity While deriving the taxonomy of DVs mistakesin classification arising from subjectiveness may have introducedunexpected coding Tomitigate this threat we followed a setmethod-ology merged coding results and performed conflict resolutionConstruct Validity In our initial study (Sec 3) a threat to con-struct validity arises in the form of themanner in which coders wereexposed to presentation failures To mitigate this threat designersfrom our industrial partner manually annotated the screen pairsin order to clearly illustrate the affected GCs on the screen In ourevaluation of Gvt threats arise from our method of DV injectionusing the synthetic fault injection tool However we designed thistool to inject faults based upon both the type and distribution offaults from our DV taxonomy to mitigate this threatExternal Validity In our initial study related to the DV taxonomywe utilized a dataset from a single (albeit large) company withexamples across several different applications and screens There isthe potential that this may not generalize to other industrial mobileapplication development environments and platforms or mobile appdevelopment in general However given the relatively consistentdesign paradigms of mobile apps we expect the categories andthe sub-categories within the taxonomy to hold although it ispossible that the distribution across these categories may varyacross application development for different domains In Study 3we surveyed employees at a single (though large) company andfindings may differ in similar studies at other companies

8 RELATEDWORKWeb Presentation Failures The work most closely related to ourapproach are approaches that aim at detecting classifying and fixingpresentation failures in web applications [30ndash32 42] In comparisonto these approaches Gvt also performs detection and localizationof presentation failures but is the first to do so for mobile appsIn addition to the engineering challenges associated with buildingan approach to detect presentation failures in the mobile domain(eg collection and processing of GUI-related data) Gvt is the firstapproach to leverage metadata from software mock-up artifacts(eg Marketch) to perform GC matching based upon the spatialinformation collected from both mock-ups and dynamic applicationscreens allowing for precise detection of the different types of DVsdelineated in our industrial DV taxonomy Gvt is also the first toapply the processes of CQ CH analysis and B-PID toward detectingdifferences in the content and color of icons and images displayedin mobile apps Gvt also explicitly identifies and reports differentfaulty properties (such as errors in component location or text)Cross Browser Testing Approaches for XBT (or cross browsertesting) by Roy Choudhry et al [18 42 43] examine and automati-cally report differences in web pages rendered in multiple browsersThese approaches are currently not directly applicable to mock-updriven development for mobile appsVisualGUI Testing A concept known as Visual GUI Testing (VGT)aims to test certain visual aspects of a software applicationrsquos GUIas well as the underlying functional properties To accomplish thisvisual GUI testing usually executes actions on a target applicationsin order to exercise app functionality [13 14 23 38] In contrastto these approaches Gvt is designed to apply to mobile-specificDVs is tailored for the mock-up driven development practice andis aimed only at verifying visual properties of a mobile apprsquos GUIOther Approaches There are other approaches and techniquesthat related to identifying problems or differences with GUIs ofmobile apps Xie et al introduced GUIDE [47] a tool for GUI differ-encing between successive releases of GUIs for an app by matchingcomponents between GUI-hierarchies Gvt utilizes a matching pro-cedure for leaf node components as direct tree comparisons are notpossible in the context of mock-up driven development There hasalso been both commercial and academic work related to graphi-cal software built specifically for creating high-fidelity mobile appmock-ups or mockups that encode information for automated cre-ation of code for a target platform [4 8 9 34] However such toolstend to either impose too many restrictions on designers or do notallow for direct creation of code thus DVs still persist in practice9 CONCLUSION amp FUTUREWORKIn this paper we have formalized the problem of detecting designviolations in mobile apps and derived a taxonomy of design viola-tions based on a robust industrial dataset We presented Gvt anapproach for automatically detecting classifying and reporting de-sign violations in mobile apps and conducted a wide ranging studythat measured performance utility and industrial applicability ofthis tool Our results indicate that Gvt is effective in practice offersutility for developers and is applicable in industrial contexts

ACKNOWLEDGMENTSThe authors would like to thank Kebing Xie Roozbeh Farahbodand the developers and designers at Huawei for their support

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden

REFERENCES[1] Adobe photoshop httpwwwphotoshopcom[2] Android uiautomator httpdeveloperandroidcomtoolshelpuiautomator

indexhtml[3] Apple app store httpswwwapplecomiosapp-store[4] Fuild-ui httpswwwfluiduicom[5] Google play store httpsplaygooglecomstorehl=en[6] The marketch plugin for sketch httpsgithubcomtudou527marketch[7] Mobile apps What consumers really need and want httpsinfodynatracecom

rscompuwareimagesMobileAppSurveyReportpdf[8] Mockupio httpsmockupioabout[9] Protoio httpsprotoio[10] The sketch design tool httpswwwsketchappcom[11] Sketch extensions httpswwwsketchappcomextensions[12] Why your apprsquos ux is more important than you think httpwwwcodemagcom

Article1401041[13] E Aleacutegroth and R Feldt On the long-term use of visual gui testing in industrial

practice a case study Empirical Software Engineering 22(6)2937ndash2971 Dec 2017[14] E Alegroth Z Gao R Oliveira and A Memon Conceptualization and evaluation

of component-based testing unified with visual gui testing An empirical studyIn 2015 IEEE 8th International Conference on Software Testing Verification andValidation (ICST) pages 1ndash10 April 2015

[15] G Bavota M Linares-Vaacutesquez C Bernal-Caacuterdenas M Di Penta R Oliveto andD Poshyvanyk The impact of api change- and fault-proneness on the user ratingsof android apps Software Engineering IEEE Transactions on 41(4)384ndash407 April2015

[16] J Brooke SUS A quick and dirty usability scale In P W Jordan B WeerdmeesterA Thomas and I L Mclelland editors Usability evaluation in industry Taylorand Francis London 1996

[17] K Charmaz Constructing Grounded Theory SAGE Publications Inc 2006[18] S R Choudhary M R Prasad and A Orso Crosscheck Combining crawling and

differencing to better detect cross-browser incompatibilities in web applicationsIn Proceedings of the 2012 IEEE Fifth International Conference on Software TestingVerification and Validation ICST rsquo12 pages 171ndash180 Washington DC USA 2012IEEE Computer Society

[19] A Ciurumelea A SchaufelbAtildeČAcircijhl S Panichella and H C Gall Analyzingreviews and code of mobile apps for better release planning In 2017 IEEE 24th In-ternational Conference on Software Analysis Evolution and Reengineering (SANER)pages 91ndash102 Feb 2017

[20] A Di Sorbo S Panichella C V Alexandru J Shimagaki C A Visaggio G Can-fora and H C Gall What would users change in my app summarizing appreviews for recommending software changes In Proceedings of the 2016 24thACM SIGSOFT International Symposium on Foundations of Software EngineeringFSE 2016 pages 499ndash510 New York NY USA 2016 ACM

[21] K Holl and F Elberzhager A mobile-specific failure classification and its usageto focus quality assurance In 2014 40th EUROMICRO Conference on SoftwareEngineering and Advanced Applications pages 385ndash388 Aug 2014

[22] G Hu X Yuan Y Tang and J Yang Efficiently effectively detecting mobileapp bugs with appdoctor In Proceedings of the Ninth European Conference onComputer Systems EuroSys rsquo14 pages 181ndash1815 New York NY USA 2014ACM

[23] A Issa J Sillito and V Garousi Visual testing of graphical user interfaces Anexploratory study towards systematic definitions and approaches In 2012 14thIEEE International Symposium on Web Systems Evolution (WSE) pages 11ndash15 Sept2012

[24] N Jones Seven best practices for optimizing mobile testing efforts TechnicalReport G00248240 Gartner

[25] K Kuusinen and T Mikkonen Designing user experience for mobile apps Long-term product owner perspective In 2013 20th Asia-Pacific Software EngineeringConference volume 1 of APSECrsquo13 pages 535ndash540 Dec 2013

[26] V Lelli A Blouin and B Baudry Classifying and qualifying gui defects In 2015IEEE 8th International Conference on Software Testing Verification and Validation(ICST) pages 1ndash10 April 2015

[27] M Linares-Vaacutesquez G Bavota C Bernal-Caacuterdenas M Di Penta R Oliveto andD Poshyvanyk Api change and fault proneness A threat to the success ofandroid apps In Proceedings of the 2013 9th Joint Meeting on Foundations ofSoftware Engineering ESECFSErsquo13 pages 477ndash487 New York NY USA 2013ACM

[28] M Linares-Vaacutesquez G Bavota M D Penta R Oliveto and D Poshyvanyk Howdo API changes trigger Stack Overflow discussions a study on the android SDK

In Proceedings of the 22nd International Conference on Program ComprehensionICPCrsquo14 pages 83ndash94 2014

[29] M Linares-Vaacutesquez G Bavota M Tufano K Moran M Di Penta C VendomeC Bernal-Caacuterdenas and D Poshyvanyk Enabling mutation testing for androidapps In Proceedings of the 2017 11th Joint Meeting on Foundations of SoftwareEngineering ESECFSE 2017 pages 233ndash244 New York NY USA 2017 ACM

[30] S Mahajan A Alameer P McMinn and W G Halfond Automated repair oflayout cross browser issues using search-based techniques In InternationalConference on Software Testing and Analysis ISSTArsquo17 2017

[31] S Mahajan and W G J Halfond Detection and localization of html presentationfailures using computer vision-based techniques In Proceedings of the 8th IEEEInternational Conference on Software Testing Verification and Validation ICSTrsquo15April 2015

[32] S Mahajan B Li P Behnamghader and W G Halfond Using visual symptomsfor debugging presentation failures in web applications In Proceeding of the9th IEEE International Conference on Software Testing Verification and Validation(ICST) ICSTrsquo16 April 2016

[33] T McDonnell B Ray andM Kim An empirical study of api stability and adoptionin the android ecosystem In Proceedings of the 2013 International Conference onSoftware Maintenance ICSMrsquo13 pages 70ndash79 2013

[34] J Meskens K Luyten and K Coninx Plug-and-design Embracing mobiledevices as part of the design environment In Proceedings of the 1st ACM SIGCHISymposium on Engineering Interactive Computing Systems EICS rsquo09 pages 149ndash154 New York NY USA 2009 ACM

[35] K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk Gvt onlineappendix httpwwwandroid-dev-toolscomgvt

[36] P Morville User experience design httpsemanticstudioscomuser_experience_design

[37] B Myers Challenges of hci design and implementation interactions 1(1)73ndash83Jan 1994

[38] B N Nguyen B Robbins I Banerjee and A Memon Guitar An innovativetool for automated testing of gui-driven software Automated Software Engg21(1)65ndash105 Mar 2014

[39] T A Nguyen and C Csallner Reverse engineering mobile application userinterfaces with REMAUI In Proceedings of the 2015 30th IEEEACM Interna-tional Conference on Automated Software Engineering ASErsquo15 pages 248ndash259Washington DC USA 2015 IEEE Computer Society

[40] F PalombaM Linares-Vaacutesquez G Bavota R Oliveto M D Penta D Poshyvanykand A D Lucia User reviews matter tracking crowdsourced reviews to supportevolution of successful apps In 2015 IEEE International Conference on SoftwareMaintenance and Evolution (ICSME) pages 291ndash300 Sept 2015

[41] F Palomba P Salza A Ciurumelea S Panichella H Gall F Ferrucci andA De Lucia Recommending and localizing change requests for mobile appsbased on user reviews In Proceedings of the 39th International Conference onSoftware Engineering ICSE rsquo17 pages 106ndash117 Piscataway NJ USA 2017 IEEEPress

[42] S Roy Choudhary M R Prasad and A Orso X-pert Accurate identification ofcross-browser issues in web applications In Proceedings of the 2013 InternationalConference on Software Engineering ICSE rsquo13 pages 702ndash711 Piscataway NJUSA 2013 IEEE Press

[43] S Roy Choudhary H Versee and A Orso Webdiff Automated identification ofcross-browser issues in web applications In Proceedings of the 2010 IEEE Interna-tional Conference on Software Maintenance ICSM rsquo10 pages 1ndash10 WashingtonDC USA 2010 IEEE Computer Society

[44] T Silva da Silva A Martin F Maurer and M Silveira User-centered design andagile methods A systematic review In Proceedings of the 2011 Agile ConferenceAGILE rsquo11 pages 77ndash86 Washington DC USA 2011 IEEE Computer Society

[45] K-J Stol P Ralph and B Fitzgerald Grounded theory in software engineeringresearch A critical review and guidelines In Proceedings of the 38th InternationalConference on Software Engineering ICSE rsquo16 pages 120ndash131 New York NY USA2016 ACM

[46] A B Tucker Computer Science Handbook Second Edition Chapman amp HallCRC2004

[47] Q Xie M Grechanik C Fu and C Cumby Guide A gui differentiator In 2009IEEE International Conference on Software Maintenance ICSMrsquo09

[48] H Yee S Pattanaik and D P Greenberg Spatiotemporal sensitivity and visualattention for efficient rendering of dynamic environments ACM Trans Graph20(1)39ndash65 Jan 2001

[49] C Zeidler C Lutteroth W Stuerzlinger and G Weber Evaluating Direct Manip-ulation Operations for Constraint-Based Layout pages 513ndash529 Springer BerlinHeidelberg Berlin Heidelberg 2013

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden

implementation screenshots (iv) links to affected lines of applica-tion source code (v) color information extracted from the CH forGCs identified to have color mismatches and (vi) the differenceimage generated by PID The source code links are generated bymatching the ids extracted from the uiautomator information backto their declarations in the layout xml files in the source code (egthose located in the res directory of an apprsquos source code) Weprovide examples of generated reports in our online appendix [35]45 Implementation amp Industrial CollaborationOur implementation of Gvt was developed in Java with a SwingGUI In addition to running the Gvt analysis the tool executableallows for one-click capture of uiautomator files and screenshotsfrom a connected device or emulator Several acceptance tests ofmock-upimplementation screen pairs with pre-existing violationsfrom apps under development within our industrial partner wereused to guide the development of the tool 12 Periodic releases ofbinaries for bothWindows and Mac were made to deploy the tool todesigners and developers within the company The authors of thispaper held regular bi-weekly meetings with members of the designand development teams to plan features and collect feedback

Using the acceptance tests and feedback from our collaboratorswe tuned the various thresholds and parameters of the tool for bestperformance The PID algorithm settings were tuned for sensitivityto capture subtle visual inconsistencies which are then later filteredthrough additional CV techniques F was set to 45 L was set to100cdm2 and C was set to 1 The GC-Matching Threshold (MC)was set to 18th the screen width of a target device the DT fordetermining suspicious GCs was set to 20 The LT was set to 5pixels (based on designer preference) the CT which determinesthe degree to which colors must match for color-based DVs was setto 85 and finally the IDT was set to 20 Gvt allows for a userto change these settings if desired additionally users are capableof defining areas of dynamic content (eg loaded from networkactivity) which should be ignored by the Gvt analysis5 DESIGN OF THE EXPERIMENTSTo evaluate Gvtrsquos performance usefulness and applicability weperform three complimentary studies answering the following RQs

bull RQ1 How well does Gvt perform in terms of detecting andclassifying design violations

bull RQ2 What utility can Gvt provide from the viewpoint ofAndroid developers

bull RQ3What is the industrial applicability of Gvt in terms ofimproving the mobile application development workflow

RQ1 and RQ2 focus on quantitatively measuring the performanceof Gvt and the utility it provides to developers through a controlledempirical and a user study respectively RQ3 reports the results ofa survey and semi-structured interviews with our collaboratorsaimed at investigating the industrial applicability of Gvt51 Study 1 Gvt Effectiveness amp PerformanceThe goal of the first study is to quantitatively measure Gvt in termsof its precision and recall in both detecting and classifying DVs

511 Study Context To carry out a controlled quantitativestudy we manually reverse engineered Sketch mockups for tenscreens for eight of the most popular apps on Google Play To de-rive this set we downloaded the top-10 apps from each category on

the Google-Play store removing the various categories correspond-ing to games (as these have non-standard GUI-components thatGvt does not support) We then randomly sampled one app fromeach of the remaining 33 categories eliminating duplicates (sinceapps can belong to more than one category) We then manuallycollected screenshots and uiautomator files from two screens foreach application using a Nexus 5 attempting to capture the ldquomainrdquoscreen that a user would typically interact with and one secondaryscreen Using the uiautomator files we generated cropped screen-shots of all the leaf nodes components for each screen of the appFrom these we were able generate 10 screens from 8 applicationsthat successfully ran through Gvt without any reported violations

512 Synthetic DV Injection With a set of correct mock-ups cor-responding to implementation screens in an app we needed a suit-able method to introduce DVs into our subjects To this end we con-structed a synthetic DV injection tool that modifies the uiautomator

xml files and corresponding screenshots in order to introduce de-sign violations from our taxonomy presented in Sec 3 The toolis composed of two components (i) an XML Parser that reads andextracts components from the screen then (ii) a Violation Generatorthat randomly selects components and injects synthetic violationsWe implemented injection for the following types of DVsLocation Violation The component is moved either horizontallyvertically or in both directions within the same container Howeverthemaximum distance from the original point is limited by a quarterof the width of the screen size This was based on the severity ofLayout Violations in our study described in Section 3 In order togenerate the image we cropped the component and moved it to thenew location replacing all the original pixels by the most prominentcolor from the surroundings in the original locationSize Violation The component size either increases or decreasesby 20 of the original size For instances where the component sizedecreases we replaced all the pixels by the most prominent colorfrom the surroundings of the original sizeMissing Component Violation This violation removes a leafcomponent from the screen replacing the original pixels by themost prominent color from the surrounding backgroundImage Violation We perturb 40 of the pixels in an image byrandomly generating an RGB value for the pixels affectedImage Color Violation This rule perturbs the color of an imageby shifting the hue of image colors by 30degComponent Color Violation This uses the same process as forImage Color Violations but we change the color by 180degFont Violation This violation randomly selects a font from theset of Arial Comic Sans MS Courier Roboto or Times Roman andapplies it to a TextView componentFont Color Violation changes the text color of a TextView com-ponent We extracted the text color using CH analysis then wechanged the color using same strategy as for Image Color Violations

513 Study Methodology In injecting the synthetic faults wetook several measures to simulate the creation of realistic faultsFirst we delineated 200 different types of design violations accord-ing to the distribution defined in our DV taxonomy in Sec 3 Wethen created a pool of 100 screens by creating random copies ofthe both the uiautomator xml files and screenshots from our initialset of 10 screens We then used the synthetic DV injection tool toseed faults into the pool of 100 screens according to the following

ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk

criteria (i) No screen can contain more than 3 injected DVs (ii)each GC should have a maximum of 1 DV injected and (iii) Eachscreen must have at least 1 injected DV After the DVs were seededeach of the 100 screens and 200 DVs were manually inspected forcorrectness Due to the random nature of the tool a small num-ber of erroneous DVs were excluded and regenerated during thisprocess (eg color perturbed to perceptually similar color) Thebreakdown of injected DVs is shown in Figure 4 and the full datasetwith description is included in our online appendix [35]

Once the final set of screens with injected violations was derivedwe ran Gvt across these subjects and measured four metrics (i)detection precision (DP ) (ii) classification precision (CP ) (iii) recall(R) and (iv) execution time per screen (ET ) We make a distinctionbetween detection and classification in our dataset because it is pos-sible thatGvt is capable of detecting but misclassifying a particularDV (eg an image color DV misclassified as an incorrect image DV )DP CP and R were measured according to the following formulas

DP CP =Tp

Tp + FpR =

Tp

Tp + Fn(6)

where for DP Tp represent injected design violations that weredetected and for CP Tp represents injected violations that wereboth detected and classified correctly In each case Fp correspondto detected DVs that were either not injected or misclassified ForRecallTp represents injected violations that were correctly detectedand Fn represents injected violations that were not detected Tocollect these measures two authors manually examined the reportsfrom Gvt in order to collect the metrics

52 Study 2 Gvt UtilitySince the ultimate goal of an approach like Gvt is to improvethe workflow of developers the goal of this second study is tomeasure the utility (ie benefit) that Gvt provides to developersby investigating two phenomena (i) The accuracy and effort ofdevelopers in detecting and classifying DVs and (ii) the perceivedutility of Gvt reports in helping to identify and resolve DVs

521 Study Context We randomly derived two sets of screensto investigate the two phenomena outlined above First we ran-domly sampled two mutually exclusive sets of 25 and 20 screensrespectively from the 100 used in Study 1 ensuring at least oneinstance of each type of DV was included in the set This resulted inboth sets of screens containing 40 design violations in total The cor-rect mockup screenshot corresponding to each screen sampled fromthe study were also extracted creating pairs of ldquocorrect mockupand ldquoincorrect implementation screenshots 10 participants withat least 5 years of Android development experience were contactedvia email to participate in the survey

522 Study Methodology We created an online survey withfour sections In the first section participants were given back-ground information regarding the definition of DVs and the differ-ent types of DVs derived in our taxonomy In the second sectionparticipants were asked about demographic information such asprogramming experience and education level In the third sectioneach participant was exposed to 5 mock-up implementation screenpairs (displayed side by side on the survey web page) and asked toidentify any observed design violations Descriptions of the DVswere given at the top of this page for reference For each screenpair participants were presented with a dropdown menu to select

80

85

90

95

100

Font

Colo

r(18)

Font

Styl

e(18

)

Imag

e Colo

r(22)

Imag

e(16

)

Miss

ing G

C(26)

Size(1

8)

Text

Conte

nt(1

4)

Trans

lation

(68)

DV Type

Per

cent

age

MetricCPDPR

Figure 4 Study 1 - Detection Precision (DP ) ClassificationPrecision (CP ) and Recall (R)a type for an observed DV and a text field to describe the errorin more detail For each participant one of the 5 mock-up screenswas a control containing no injected violations The 25 screenswere assigned to participants such that each screen was observedby two participants and the order of the screens presented to eachparticipant was randomized to avoid bias To measure the effective-ness of participants in detecting and describing DVs we leveragethe DP CP and R metrics introduced in Study 1 In the fourth sec-tion participants were presented with two screen pairs from thesecond set of 20 sampled from the user study as well as the Gvtreports for these screens Participants were then asked to answer5 user-preferences (UP) and 5 user experience (UX) questions aboutthese reports which are presented in the following section The UPquestions were developed according to the user experience hon-eycomb originally developed by Morville [36] and were posed toparticipants as free form text entry questions We forgo a discussionof the free-form question responses due to space limitations butwe offer full anonymized participant responses in our online ap-pendix [35] We derived the Likert scale-based UX questions usingthe SUS usability scale by Brooke [16]

53 Study 3 Industrial Applicability of GvtThe goal of this final study is determine industrial applicability ofGvt To investigate this we worked with Huawei to collect twosources of information (i) the results of a survey sent to designersand developers who used Gvt in their daily developmentdesignworkflow and (ii) semi-structured interviews with both design anddevelopment managers whose teams have adopted the use of Gvt

531 Study Context ampMethodology We created a survey posingquestions related to the applicability of Gvt to industrial designersand developers These questions are shown in Fig 7 The semi-structured interviews were conducted in Chinese recorded andthen later translated During the interview managers were askedto respond to four questions related to the impact and performanceof the tool in practice We include discussions of the responses inSection 6 and stipulate full questions in our appendix

6 EMPIRICAL RESULTS61 Study 1 Results GVT PerformanceThe results of Study 1 are shown in Figure 4 This figure shows theaverageDP CP and R for each type of seeded violation over the 200seeded faults and the number of faults seeded into each category(following the distributions of our derived taxonomy) are shown onthe x-axis Overall these results are extremely encouraging withthe overall DP achieving 994 the average CP being 984 and

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden

CPDP

R

000 025 050 075 100

Figure 5 Study 2 - Developer CP DP and R

I found these reports unnecessarily complex

I found these reports very cumbersome to read

I think that I would like to use these reports frequently for Identifying Presentation Issues

I thought these reports were very useful for accurately identifying Presentation Errors

These reports are easy to readunderstand

SD D N A SA

Figure 6 Study 2 - UXQuestion Responses SD=Strongly Dis-agree D=Disagree N=Neutral A=Agree SA=Strongly Agree

the average R reaching 965 This illustrates that Gvt is capableof detecting seeded faults designed to emulate both the type anddistribution of DVs encountered in industrial settings While Gvtachieved at least 85 precision for each type of seeded DV it per-formed worse on some types of violations compared to others Forinstance Gvt saw its lowest precision values for the Font-Style andFont-Color violations typically due to the fact that the magnitudeof perturbation for the color or font type was not large enough tosurpass the Color or Image Difference Thresholds (CT amp IDT ) Gvttook 368 mins to process and generate reports for the set of 100screens with injected DVs or 22 sec per screen pair This executioncost was generally acceptable by our industrial collaborators

62 Study 2 Results GVT UtilityThe DP CP and R results representing the Android developersability to correctly detect and classify DVs is shown in Figure 5as box-plots across all 10 participants Here we found CP=DP aswhen a user misclassified violations they also did not detect themAs this figure shows the Android developers generally performedmuch worse compared to Gvt achieving an average CP of underasymp 60 and an average R of asymp 50 The sources of this performanceloss for the study participants compared to Gvt was fourfold (i)participants tended to report minor acceptable differences in fontsacross the examples (despite the instructions clearly stating notto report such violations) (ii) users tended to attribute more thanone DV to a single component specifically for font style and fontcolor violations despite instructions to report only one (iii) userstended to misclassify DVs based on the provided categories (egclassifying a layout DV for a Text GC as an incorrect text DV ) and(iv) participants missed reporting many of the injected DVs lead-ing to the low recall numbers These results indicate that at thevery least developers can struggle to both detect and classify DVsbetween mock-up and implementation screen pairs signaling theneed for an automated system to check for DVs before implementedapps are sent to a UIUX team for auditing This result confirmsthe notion that developers may not be as sensitive to small DVsin the GUI as the designers who created the GUI specificationsFurthermore this finding is notable because as part of the iterativeprocess of resolving design violations designers must communicateto developers DVs and developers must recognize and understandthese DVs in order to properly resolve them This process is oftencomplicated due to ambiguous descriptions of DVs from designers

The GVT allowed for better transfer of the design from mockminusups to the implementation of the app

The GVT has helped you to reduce the time required for verifying design violations

The GVT is able to accurately report existing design violations in productionminusquality applications

The GVT tool helped my team (designimplementation) communicate with other teams (implementationdesign)

regarding GUI design violations

Using the GUIminusVerification tool (GVT) helped to improve the quality of mobile

applications produced by industrial partner

SD D N A SA

Figure 7 Study 3 - Applicability Questions SD=Strongly Dis-agree D=Disagree N=Neutral A=Agree SA=Strongly Agree

to developers or developers disagreeing with designers over the ex-istence or type of a DV In contrast to this fragmented process Gvtprovides clear unambiguous reports that facilitate communicationbetween designers and developers

Figure 6 illustrates the responses to the likert based UX ques-tions and the results are quite encouraging In general participantsfound that the reports from Gvt were easy to read useful for iden-tifying DVs and indicated that they would like to use the reports foridentifying DVs Participants also indicated that the reports werenot unnecessarily complex or difficult to read We asked the partic-ipants about their preferences for the Gvt reports as well askingabout the most and least useful information in the reports Everysingle participant indicated that the highlighted annotations on thescreenshots in the report were the most useful element Whereasmost users tended to dislike the PID output included at the bottomof the report citing this information as difficult to comprehend

63 Study 3 Results Industrial ApplicabilityThe results for the applicability questions asked to 20 designersand developers who use Gvt in their daily activities is shown inFigure 7 A positive outcome for each of these statements correlatesto responses indicating that developers ldquoagreerdquo or ldquostrongly agreerdquoThe results of this study indicate a weak agreement of developersfor these statements indicating that while Gvt is generally appli-cable there are some drawbacks that prevented developers anddesigners from giving the tool unequivocal support We explorethese drawbacks by conducting semi-structured interviews

In conducting the interviews one of the authors asked the ques-tions presented in Figure 7 to 3 managers (2 from UIUX teams and1 from a Front-End development team) When asked whether Gvtcontributed to an increased quality of mobile applications at thecompany all three managers tended to agree that this was the caseFor instance one of the design managers stated ldquoCertainly yes Thetool is the industryrsquos first and the other designer manager addedldquoWhen the page is more complicated the tool is more helpful

When asked about the overall performance and accuracy of thetool in detecting DVs the manager from the implementation teamadmitted that the current detection performance of the tool is goodbut suggested that dynamic detection of some components mayimprove it stating ldquo[DVs ] can be detected pretty well [but the toolis] not very flexible For example a switch component in the designis open but the switch is off in the implementation He suggestedthat properly handling cases such as this would make the toolmore useful from a developers perspective One of the design teammanagers held a similar view stating that ldquoCurrently most errorsare layout errors so tool is accurate Static components are basicallydetected [but] maybe the next extension should focus on dynamic

ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk

components While the current version of the Gvt allows for theexclusion of regions with dynamic components it is clear that bothdesign and development teams would appreciate proper detectionof DVs for dynamic components Additionally two of the managerscommented on the ldquorigidityrdquo of the Gvtrsquos current interface andexplained that a more streamlined UI would help improve its utility

When asked about whether Gvt improved communication be-tween the design and development teams the development teammanager felt that while the tool has not improved communica-tion yet it did have the potential to do so ldquoAt present there is no[improvement] but certainly there is the potential possibility Thedesign managers generally stated that the tool has helped withcommunication particularly in clarifying subtle DVs that may havecaused arguments between teams in the past ldquoIf you consider thetime savings on discussion and arguments between the two teamsthis tool saves us a lot of time Another designer indicated that thetool is helpful at describing DVs to developers who may not be ableto recognize them with the naked eye ldquoWe found that the tool canindeed detect something that the naked eye cannot While there arecertainly further refinements that can be made to Gvt it is clearthat the tool has begun to have a positive impact of the developmentof mobile apps and as the tool evolves within the company shouldallow for continued improvements in quality and time saved

7 LIMITATIONS amp THREATS TO VALIDITYLimitations While we have illustrated thatGvt is applicable in anindustrial setting the tool is not without its limitations Currentlythe tool imposes lightweight restrictions on designers creatingSketch mock-ups chief among these being the requirement thatbounding boxes of components do not overlap Currently Gvt willtry to resolve such cases during the GUI-Comprehension stage usingan Intersection over union (IOU) metricInternal Validity While deriving the taxonomy of DVs mistakesin classification arising from subjectiveness may have introducedunexpected coding Tomitigate this threat we followed a setmethod-ology merged coding results and performed conflict resolutionConstruct Validity In our initial study (Sec 3) a threat to con-struct validity arises in the form of themanner in which coders wereexposed to presentation failures To mitigate this threat designersfrom our industrial partner manually annotated the screen pairsin order to clearly illustrate the affected GCs on the screen In ourevaluation of Gvt threats arise from our method of DV injectionusing the synthetic fault injection tool However we designed thistool to inject faults based upon both the type and distribution offaults from our DV taxonomy to mitigate this threatExternal Validity In our initial study related to the DV taxonomywe utilized a dataset from a single (albeit large) company withexamples across several different applications and screens There isthe potential that this may not generalize to other industrial mobileapplication development environments and platforms or mobile appdevelopment in general However given the relatively consistentdesign paradigms of mobile apps we expect the categories andthe sub-categories within the taxonomy to hold although it ispossible that the distribution across these categories may varyacross application development for different domains In Study 3we surveyed employees at a single (though large) company andfindings may differ in similar studies at other companies

8 RELATEDWORKWeb Presentation Failures The work most closely related to ourapproach are approaches that aim at detecting classifying and fixingpresentation failures in web applications [30ndash32 42] In comparisonto these approaches Gvt also performs detection and localizationof presentation failures but is the first to do so for mobile appsIn addition to the engineering challenges associated with buildingan approach to detect presentation failures in the mobile domain(eg collection and processing of GUI-related data) Gvt is the firstapproach to leverage metadata from software mock-up artifacts(eg Marketch) to perform GC matching based upon the spatialinformation collected from both mock-ups and dynamic applicationscreens allowing for precise detection of the different types of DVsdelineated in our industrial DV taxonomy Gvt is also the first toapply the processes of CQ CH analysis and B-PID toward detectingdifferences in the content and color of icons and images displayedin mobile apps Gvt also explicitly identifies and reports differentfaulty properties (such as errors in component location or text)Cross Browser Testing Approaches for XBT (or cross browsertesting) by Roy Choudhry et al [18 42 43] examine and automati-cally report differences in web pages rendered in multiple browsersThese approaches are currently not directly applicable to mock-updriven development for mobile appsVisualGUI Testing A concept known as Visual GUI Testing (VGT)aims to test certain visual aspects of a software applicationrsquos GUIas well as the underlying functional properties To accomplish thisvisual GUI testing usually executes actions on a target applicationsin order to exercise app functionality [13 14 23 38] In contrastto these approaches Gvt is designed to apply to mobile-specificDVs is tailored for the mock-up driven development practice andis aimed only at verifying visual properties of a mobile apprsquos GUIOther Approaches There are other approaches and techniquesthat related to identifying problems or differences with GUIs ofmobile apps Xie et al introduced GUIDE [47] a tool for GUI differ-encing between successive releases of GUIs for an app by matchingcomponents between GUI-hierarchies Gvt utilizes a matching pro-cedure for leaf node components as direct tree comparisons are notpossible in the context of mock-up driven development There hasalso been both commercial and academic work related to graphi-cal software built specifically for creating high-fidelity mobile appmock-ups or mockups that encode information for automated cre-ation of code for a target platform [4 8 9 34] However such toolstend to either impose too many restrictions on designers or do notallow for direct creation of code thus DVs still persist in practice9 CONCLUSION amp FUTUREWORKIn this paper we have formalized the problem of detecting designviolations in mobile apps and derived a taxonomy of design viola-tions based on a robust industrial dataset We presented Gvt anapproach for automatically detecting classifying and reporting de-sign violations in mobile apps and conducted a wide ranging studythat measured performance utility and industrial applicability ofthis tool Our results indicate that Gvt is effective in practice offersutility for developers and is applicable in industrial contexts

ACKNOWLEDGMENTSThe authors would like to thank Kebing Xie Roozbeh Farahbodand the developers and designers at Huawei for their support

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden

REFERENCES[1] Adobe photoshop httpwwwphotoshopcom[2] Android uiautomator httpdeveloperandroidcomtoolshelpuiautomator

indexhtml[3] Apple app store httpswwwapplecomiosapp-store[4] Fuild-ui httpswwwfluiduicom[5] Google play store httpsplaygooglecomstorehl=en[6] The marketch plugin for sketch httpsgithubcomtudou527marketch[7] Mobile apps What consumers really need and want httpsinfodynatracecom

rscompuwareimagesMobileAppSurveyReportpdf[8] Mockupio httpsmockupioabout[9] Protoio httpsprotoio[10] The sketch design tool httpswwwsketchappcom[11] Sketch extensions httpswwwsketchappcomextensions[12] Why your apprsquos ux is more important than you think httpwwwcodemagcom

Article1401041[13] E Aleacutegroth and R Feldt On the long-term use of visual gui testing in industrial

practice a case study Empirical Software Engineering 22(6)2937ndash2971 Dec 2017[14] E Alegroth Z Gao R Oliveira and A Memon Conceptualization and evaluation

of component-based testing unified with visual gui testing An empirical studyIn 2015 IEEE 8th International Conference on Software Testing Verification andValidation (ICST) pages 1ndash10 April 2015

[15] G Bavota M Linares-Vaacutesquez C Bernal-Caacuterdenas M Di Penta R Oliveto andD Poshyvanyk The impact of api change- and fault-proneness on the user ratingsof android apps Software Engineering IEEE Transactions on 41(4)384ndash407 April2015

[16] J Brooke SUS A quick and dirty usability scale In P W Jordan B WeerdmeesterA Thomas and I L Mclelland editors Usability evaluation in industry Taylorand Francis London 1996

[17] K Charmaz Constructing Grounded Theory SAGE Publications Inc 2006[18] S R Choudhary M R Prasad and A Orso Crosscheck Combining crawling and

differencing to better detect cross-browser incompatibilities in web applicationsIn Proceedings of the 2012 IEEE Fifth International Conference on Software TestingVerification and Validation ICST rsquo12 pages 171ndash180 Washington DC USA 2012IEEE Computer Society

[19] A Ciurumelea A SchaufelbAtildeČAcircijhl S Panichella and H C Gall Analyzingreviews and code of mobile apps for better release planning In 2017 IEEE 24th In-ternational Conference on Software Analysis Evolution and Reengineering (SANER)pages 91ndash102 Feb 2017

[20] A Di Sorbo S Panichella C V Alexandru J Shimagaki C A Visaggio G Can-fora and H C Gall What would users change in my app summarizing appreviews for recommending software changes In Proceedings of the 2016 24thACM SIGSOFT International Symposium on Foundations of Software EngineeringFSE 2016 pages 499ndash510 New York NY USA 2016 ACM

[21] K Holl and F Elberzhager A mobile-specific failure classification and its usageto focus quality assurance In 2014 40th EUROMICRO Conference on SoftwareEngineering and Advanced Applications pages 385ndash388 Aug 2014

[22] G Hu X Yuan Y Tang and J Yang Efficiently effectively detecting mobileapp bugs with appdoctor In Proceedings of the Ninth European Conference onComputer Systems EuroSys rsquo14 pages 181ndash1815 New York NY USA 2014ACM

[23] A Issa J Sillito and V Garousi Visual testing of graphical user interfaces Anexploratory study towards systematic definitions and approaches In 2012 14thIEEE International Symposium on Web Systems Evolution (WSE) pages 11ndash15 Sept2012

[24] N Jones Seven best practices for optimizing mobile testing efforts TechnicalReport G00248240 Gartner

[25] K Kuusinen and T Mikkonen Designing user experience for mobile apps Long-term product owner perspective In 2013 20th Asia-Pacific Software EngineeringConference volume 1 of APSECrsquo13 pages 535ndash540 Dec 2013

[26] V Lelli A Blouin and B Baudry Classifying and qualifying gui defects In 2015IEEE 8th International Conference on Software Testing Verification and Validation(ICST) pages 1ndash10 April 2015

[27] M Linares-Vaacutesquez G Bavota C Bernal-Caacuterdenas M Di Penta R Oliveto andD Poshyvanyk Api change and fault proneness A threat to the success ofandroid apps In Proceedings of the 2013 9th Joint Meeting on Foundations ofSoftware Engineering ESECFSErsquo13 pages 477ndash487 New York NY USA 2013ACM

[28] M Linares-Vaacutesquez G Bavota M D Penta R Oliveto and D Poshyvanyk Howdo API changes trigger Stack Overflow discussions a study on the android SDK

In Proceedings of the 22nd International Conference on Program ComprehensionICPCrsquo14 pages 83ndash94 2014

[29] M Linares-Vaacutesquez G Bavota M Tufano K Moran M Di Penta C VendomeC Bernal-Caacuterdenas and D Poshyvanyk Enabling mutation testing for androidapps In Proceedings of the 2017 11th Joint Meeting on Foundations of SoftwareEngineering ESECFSE 2017 pages 233ndash244 New York NY USA 2017 ACM

[30] S Mahajan A Alameer P McMinn and W G Halfond Automated repair oflayout cross browser issues using search-based techniques In InternationalConference on Software Testing and Analysis ISSTArsquo17 2017

[31] S Mahajan and W G J Halfond Detection and localization of html presentationfailures using computer vision-based techniques In Proceedings of the 8th IEEEInternational Conference on Software Testing Verification and Validation ICSTrsquo15April 2015

[32] S Mahajan B Li P Behnamghader and W G Halfond Using visual symptomsfor debugging presentation failures in web applications In Proceeding of the9th IEEE International Conference on Software Testing Verification and Validation(ICST) ICSTrsquo16 April 2016

[33] T McDonnell B Ray andM Kim An empirical study of api stability and adoptionin the android ecosystem In Proceedings of the 2013 International Conference onSoftware Maintenance ICSMrsquo13 pages 70ndash79 2013

[34] J Meskens K Luyten and K Coninx Plug-and-design Embracing mobiledevices as part of the design environment In Proceedings of the 1st ACM SIGCHISymposium on Engineering Interactive Computing Systems EICS rsquo09 pages 149ndash154 New York NY USA 2009 ACM

[35] K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk Gvt onlineappendix httpwwwandroid-dev-toolscomgvt

[36] P Morville User experience design httpsemanticstudioscomuser_experience_design

[37] B Myers Challenges of hci design and implementation interactions 1(1)73ndash83Jan 1994

[38] B N Nguyen B Robbins I Banerjee and A Memon Guitar An innovativetool for automated testing of gui-driven software Automated Software Engg21(1)65ndash105 Mar 2014

[39] T A Nguyen and C Csallner Reverse engineering mobile application userinterfaces with REMAUI In Proceedings of the 2015 30th IEEEACM Interna-tional Conference on Automated Software Engineering ASErsquo15 pages 248ndash259Washington DC USA 2015 IEEE Computer Society

[40] F PalombaM Linares-Vaacutesquez G Bavota R Oliveto M D Penta D Poshyvanykand A D Lucia User reviews matter tracking crowdsourced reviews to supportevolution of successful apps In 2015 IEEE International Conference on SoftwareMaintenance and Evolution (ICSME) pages 291ndash300 Sept 2015

[41] F Palomba P Salza A Ciurumelea S Panichella H Gall F Ferrucci andA De Lucia Recommending and localizing change requests for mobile appsbased on user reviews In Proceedings of the 39th International Conference onSoftware Engineering ICSE rsquo17 pages 106ndash117 Piscataway NJ USA 2017 IEEEPress

[42] S Roy Choudhary M R Prasad and A Orso X-pert Accurate identification ofcross-browser issues in web applications In Proceedings of the 2013 InternationalConference on Software Engineering ICSE rsquo13 pages 702ndash711 Piscataway NJUSA 2013 IEEE Press

[43] S Roy Choudhary H Versee and A Orso Webdiff Automated identification ofcross-browser issues in web applications In Proceedings of the 2010 IEEE Interna-tional Conference on Software Maintenance ICSM rsquo10 pages 1ndash10 WashingtonDC USA 2010 IEEE Computer Society

[44] T Silva da Silva A Martin F Maurer and M Silveira User-centered design andagile methods A systematic review In Proceedings of the 2011 Agile ConferenceAGILE rsquo11 pages 77ndash86 Washington DC USA 2011 IEEE Computer Society

[45] K-J Stol P Ralph and B Fitzgerald Grounded theory in software engineeringresearch A critical review and guidelines In Proceedings of the 38th InternationalConference on Software Engineering ICSE rsquo16 pages 120ndash131 New York NY USA2016 ACM

[46] A B Tucker Computer Science Handbook Second Edition Chapman amp HallCRC2004

[47] Q Xie M Grechanik C Fu and C Cumby Guide A gui differentiator In 2009IEEE International Conference on Software Maintenance ICSMrsquo09

[48] H Yee S Pattanaik and D P Greenberg Spatiotemporal sensitivity and visualattention for efficient rendering of dynamic environments ACM Trans Graph20(1)39ndash65 Jan 2001

[49] C Zeidler C Lutteroth W Stuerzlinger and G Weber Evaluating Direct Manip-ulation Operations for Constraint-Based Layout pages 513ndash529 Springer BerlinHeidelberg Berlin Heidelberg 2013

ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk

criteria (i) No screen can contain more than 3 injected DVs (ii)each GC should have a maximum of 1 DV injected and (iii) Eachscreen must have at least 1 injected DV After the DVs were seededeach of the 100 screens and 200 DVs were manually inspected forcorrectness Due to the random nature of the tool a small num-ber of erroneous DVs were excluded and regenerated during thisprocess (eg color perturbed to perceptually similar color) Thebreakdown of injected DVs is shown in Figure 4 and the full datasetwith description is included in our online appendix [35]

Once the final set of screens with injected violations was derivedwe ran Gvt across these subjects and measured four metrics (i)detection precision (DP ) (ii) classification precision (CP ) (iii) recall(R) and (iv) execution time per screen (ET ) We make a distinctionbetween detection and classification in our dataset because it is pos-sible thatGvt is capable of detecting but misclassifying a particularDV (eg an image color DV misclassified as an incorrect image DV )DP CP and R were measured according to the following formulas

DP CP =Tp

Tp + FpR =

Tp

Tp + Fn(6)

where for DP Tp represent injected design violations that weredetected and for CP Tp represents injected violations that wereboth detected and classified correctly In each case Fp correspondto detected DVs that were either not injected or misclassified ForRecallTp represents injected violations that were correctly detectedand Fn represents injected violations that were not detected Tocollect these measures two authors manually examined the reportsfrom Gvt in order to collect the metrics

52 Study 2 Gvt UtilitySince the ultimate goal of an approach like Gvt is to improvethe workflow of developers the goal of this second study is tomeasure the utility (ie benefit) that Gvt provides to developersby investigating two phenomena (i) The accuracy and effort ofdevelopers in detecting and classifying DVs and (ii) the perceivedutility of Gvt reports in helping to identify and resolve DVs

521 Study Context We randomly derived two sets of screensto investigate the two phenomena outlined above First we ran-domly sampled two mutually exclusive sets of 25 and 20 screensrespectively from the 100 used in Study 1 ensuring at least oneinstance of each type of DV was included in the set This resulted inboth sets of screens containing 40 design violations in total The cor-rect mockup screenshot corresponding to each screen sampled fromthe study were also extracted creating pairs of ldquocorrect mockupand ldquoincorrect implementation screenshots 10 participants withat least 5 years of Android development experience were contactedvia email to participate in the survey

522 Study Methodology We created an online survey withfour sections In the first section participants were given back-ground information regarding the definition of DVs and the differ-ent types of DVs derived in our taxonomy In the second sectionparticipants were asked about demographic information such asprogramming experience and education level In the third sectioneach participant was exposed to 5 mock-up implementation screenpairs (displayed side by side on the survey web page) and asked toidentify any observed design violations Descriptions of the DVswere given at the top of this page for reference For each screenpair participants were presented with a dropdown menu to select

80

85

90

95

100

Font

Colo

r(18)

Font

Styl

e(18

)

Imag

e Colo

r(22)

Imag

e(16

)

Miss

ing G

C(26)

Size(1

8)

Text

Conte

nt(1

4)

Trans

lation

(68)

DV Type

Per

cent

age

MetricCPDPR

Figure 4 Study 1 - Detection Precision (DP ) ClassificationPrecision (CP ) and Recall (R)a type for an observed DV and a text field to describe the errorin more detail For each participant one of the 5 mock-up screenswas a control containing no injected violations The 25 screenswere assigned to participants such that each screen was observedby two participants and the order of the screens presented to eachparticipant was randomized to avoid bias To measure the effective-ness of participants in detecting and describing DVs we leveragethe DP CP and R metrics introduced in Study 1 In the fourth sec-tion participants were presented with two screen pairs from thesecond set of 20 sampled from the user study as well as the Gvtreports for these screens Participants were then asked to answer5 user-preferences (UP) and 5 user experience (UX) questions aboutthese reports which are presented in the following section The UPquestions were developed according to the user experience hon-eycomb originally developed by Morville [36] and were posed toparticipants as free form text entry questions We forgo a discussionof the free-form question responses due to space limitations butwe offer full anonymized participant responses in our online ap-pendix [35] We derived the Likert scale-based UX questions usingthe SUS usability scale by Brooke [16]

53 Study 3 Industrial Applicability of GvtThe goal of this final study is determine industrial applicability ofGvt To investigate this we worked with Huawei to collect twosources of information (i) the results of a survey sent to designersand developers who used Gvt in their daily developmentdesignworkflow and (ii) semi-structured interviews with both design anddevelopment managers whose teams have adopted the use of Gvt

531 Study Context ampMethodology We created a survey posingquestions related to the applicability of Gvt to industrial designersand developers These questions are shown in Fig 7 The semi-structured interviews were conducted in Chinese recorded andthen later translated During the interview managers were askedto respond to four questions related to the impact and performanceof the tool in practice We include discussions of the responses inSection 6 and stipulate full questions in our appendix

6 EMPIRICAL RESULTS61 Study 1 Results GVT PerformanceThe results of Study 1 are shown in Figure 4 This figure shows theaverageDP CP and R for each type of seeded violation over the 200seeded faults and the number of faults seeded into each category(following the distributions of our derived taxonomy) are shown onthe x-axis Overall these results are extremely encouraging withthe overall DP achieving 994 the average CP being 984 and

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden

CPDP

R

000 025 050 075 100

Figure 5 Study 2 - Developer CP DP and R

I found these reports unnecessarily complex

I found these reports very cumbersome to read

I think that I would like to use these reports frequently for Identifying Presentation Issues

I thought these reports were very useful for accurately identifying Presentation Errors

These reports are easy to readunderstand

SD D N A SA

Figure 6 Study 2 - UXQuestion Responses SD=Strongly Dis-agree D=Disagree N=Neutral A=Agree SA=Strongly Agree

the average R reaching 965 This illustrates that Gvt is capableof detecting seeded faults designed to emulate both the type anddistribution of DVs encountered in industrial settings While Gvtachieved at least 85 precision for each type of seeded DV it per-formed worse on some types of violations compared to others Forinstance Gvt saw its lowest precision values for the Font-Style andFont-Color violations typically due to the fact that the magnitudeof perturbation for the color or font type was not large enough tosurpass the Color or Image Difference Thresholds (CT amp IDT ) Gvttook 368 mins to process and generate reports for the set of 100screens with injected DVs or 22 sec per screen pair This executioncost was generally acceptable by our industrial collaborators

62 Study 2 Results GVT UtilityThe DP CP and R results representing the Android developersability to correctly detect and classify DVs is shown in Figure 5as box-plots across all 10 participants Here we found CP=DP aswhen a user misclassified violations they also did not detect themAs this figure shows the Android developers generally performedmuch worse compared to Gvt achieving an average CP of underasymp 60 and an average R of asymp 50 The sources of this performanceloss for the study participants compared to Gvt was fourfold (i)participants tended to report minor acceptable differences in fontsacross the examples (despite the instructions clearly stating notto report such violations) (ii) users tended to attribute more thanone DV to a single component specifically for font style and fontcolor violations despite instructions to report only one (iii) userstended to misclassify DVs based on the provided categories (egclassifying a layout DV for a Text GC as an incorrect text DV ) and(iv) participants missed reporting many of the injected DVs lead-ing to the low recall numbers These results indicate that at thevery least developers can struggle to both detect and classify DVsbetween mock-up and implementation screen pairs signaling theneed for an automated system to check for DVs before implementedapps are sent to a UIUX team for auditing This result confirmsthe notion that developers may not be as sensitive to small DVsin the GUI as the designers who created the GUI specificationsFurthermore this finding is notable because as part of the iterativeprocess of resolving design violations designers must communicateto developers DVs and developers must recognize and understandthese DVs in order to properly resolve them This process is oftencomplicated due to ambiguous descriptions of DVs from designers

The GVT allowed for better transfer of the design from mockminusups to the implementation of the app

The GVT has helped you to reduce the time required for verifying design violations

The GVT is able to accurately report existing design violations in productionminusquality applications

The GVT tool helped my team (designimplementation) communicate with other teams (implementationdesign)

regarding GUI design violations

Using the GUIminusVerification tool (GVT) helped to improve the quality of mobile

applications produced by industrial partner

SD D N A SA

Figure 7 Study 3 - Applicability Questions SD=Strongly Dis-agree D=Disagree N=Neutral A=Agree SA=Strongly Agree

to developers or developers disagreeing with designers over the ex-istence or type of a DV In contrast to this fragmented process Gvtprovides clear unambiguous reports that facilitate communicationbetween designers and developers

Figure 6 illustrates the responses to the likert based UX ques-tions and the results are quite encouraging In general participantsfound that the reports from Gvt were easy to read useful for iden-tifying DVs and indicated that they would like to use the reports foridentifying DVs Participants also indicated that the reports werenot unnecessarily complex or difficult to read We asked the partic-ipants about their preferences for the Gvt reports as well askingabout the most and least useful information in the reports Everysingle participant indicated that the highlighted annotations on thescreenshots in the report were the most useful element Whereasmost users tended to dislike the PID output included at the bottomof the report citing this information as difficult to comprehend

63 Study 3 Results Industrial ApplicabilityThe results for the applicability questions asked to 20 designersand developers who use Gvt in their daily activities is shown inFigure 7 A positive outcome for each of these statements correlatesto responses indicating that developers ldquoagreerdquo or ldquostrongly agreerdquoThe results of this study indicate a weak agreement of developersfor these statements indicating that while Gvt is generally appli-cable there are some drawbacks that prevented developers anddesigners from giving the tool unequivocal support We explorethese drawbacks by conducting semi-structured interviews

In conducting the interviews one of the authors asked the ques-tions presented in Figure 7 to 3 managers (2 from UIUX teams and1 from a Front-End development team) When asked whether Gvtcontributed to an increased quality of mobile applications at thecompany all three managers tended to agree that this was the caseFor instance one of the design managers stated ldquoCertainly yes Thetool is the industryrsquos first and the other designer manager addedldquoWhen the page is more complicated the tool is more helpful

When asked about the overall performance and accuracy of thetool in detecting DVs the manager from the implementation teamadmitted that the current detection performance of the tool is goodbut suggested that dynamic detection of some components mayimprove it stating ldquo[DVs ] can be detected pretty well [but the toolis] not very flexible For example a switch component in the designis open but the switch is off in the implementation He suggestedthat properly handling cases such as this would make the toolmore useful from a developers perspective One of the design teammanagers held a similar view stating that ldquoCurrently most errorsare layout errors so tool is accurate Static components are basicallydetected [but] maybe the next extension should focus on dynamic

ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk

components While the current version of the Gvt allows for theexclusion of regions with dynamic components it is clear that bothdesign and development teams would appreciate proper detectionof DVs for dynamic components Additionally two of the managerscommented on the ldquorigidityrdquo of the Gvtrsquos current interface andexplained that a more streamlined UI would help improve its utility

When asked about whether Gvt improved communication be-tween the design and development teams the development teammanager felt that while the tool has not improved communica-tion yet it did have the potential to do so ldquoAt present there is no[improvement] but certainly there is the potential possibility Thedesign managers generally stated that the tool has helped withcommunication particularly in clarifying subtle DVs that may havecaused arguments between teams in the past ldquoIf you consider thetime savings on discussion and arguments between the two teamsthis tool saves us a lot of time Another designer indicated that thetool is helpful at describing DVs to developers who may not be ableto recognize them with the naked eye ldquoWe found that the tool canindeed detect something that the naked eye cannot While there arecertainly further refinements that can be made to Gvt it is clearthat the tool has begun to have a positive impact of the developmentof mobile apps and as the tool evolves within the company shouldallow for continued improvements in quality and time saved

7 LIMITATIONS amp THREATS TO VALIDITYLimitations While we have illustrated thatGvt is applicable in anindustrial setting the tool is not without its limitations Currentlythe tool imposes lightweight restrictions on designers creatingSketch mock-ups chief among these being the requirement thatbounding boxes of components do not overlap Currently Gvt willtry to resolve such cases during the GUI-Comprehension stage usingan Intersection over union (IOU) metricInternal Validity While deriving the taxonomy of DVs mistakesin classification arising from subjectiveness may have introducedunexpected coding Tomitigate this threat we followed a setmethod-ology merged coding results and performed conflict resolutionConstruct Validity In our initial study (Sec 3) a threat to con-struct validity arises in the form of themanner in which coders wereexposed to presentation failures To mitigate this threat designersfrom our industrial partner manually annotated the screen pairsin order to clearly illustrate the affected GCs on the screen In ourevaluation of Gvt threats arise from our method of DV injectionusing the synthetic fault injection tool However we designed thistool to inject faults based upon both the type and distribution offaults from our DV taxonomy to mitigate this threatExternal Validity In our initial study related to the DV taxonomywe utilized a dataset from a single (albeit large) company withexamples across several different applications and screens There isthe potential that this may not generalize to other industrial mobileapplication development environments and platforms or mobile appdevelopment in general However given the relatively consistentdesign paradigms of mobile apps we expect the categories andthe sub-categories within the taxonomy to hold although it ispossible that the distribution across these categories may varyacross application development for different domains In Study 3we surveyed employees at a single (though large) company andfindings may differ in similar studies at other companies

8 RELATEDWORKWeb Presentation Failures The work most closely related to ourapproach are approaches that aim at detecting classifying and fixingpresentation failures in web applications [30ndash32 42] In comparisonto these approaches Gvt also performs detection and localizationof presentation failures but is the first to do so for mobile appsIn addition to the engineering challenges associated with buildingan approach to detect presentation failures in the mobile domain(eg collection and processing of GUI-related data) Gvt is the firstapproach to leverage metadata from software mock-up artifacts(eg Marketch) to perform GC matching based upon the spatialinformation collected from both mock-ups and dynamic applicationscreens allowing for precise detection of the different types of DVsdelineated in our industrial DV taxonomy Gvt is also the first toapply the processes of CQ CH analysis and B-PID toward detectingdifferences in the content and color of icons and images displayedin mobile apps Gvt also explicitly identifies and reports differentfaulty properties (such as errors in component location or text)Cross Browser Testing Approaches for XBT (or cross browsertesting) by Roy Choudhry et al [18 42 43] examine and automati-cally report differences in web pages rendered in multiple browsersThese approaches are currently not directly applicable to mock-updriven development for mobile appsVisualGUI Testing A concept known as Visual GUI Testing (VGT)aims to test certain visual aspects of a software applicationrsquos GUIas well as the underlying functional properties To accomplish thisvisual GUI testing usually executes actions on a target applicationsin order to exercise app functionality [13 14 23 38] In contrastto these approaches Gvt is designed to apply to mobile-specificDVs is tailored for the mock-up driven development practice andis aimed only at verifying visual properties of a mobile apprsquos GUIOther Approaches There are other approaches and techniquesthat related to identifying problems or differences with GUIs ofmobile apps Xie et al introduced GUIDE [47] a tool for GUI differ-encing between successive releases of GUIs for an app by matchingcomponents between GUI-hierarchies Gvt utilizes a matching pro-cedure for leaf node components as direct tree comparisons are notpossible in the context of mock-up driven development There hasalso been both commercial and academic work related to graphi-cal software built specifically for creating high-fidelity mobile appmock-ups or mockups that encode information for automated cre-ation of code for a target platform [4 8 9 34] However such toolstend to either impose too many restrictions on designers or do notallow for direct creation of code thus DVs still persist in practice9 CONCLUSION amp FUTUREWORKIn this paper we have formalized the problem of detecting designviolations in mobile apps and derived a taxonomy of design viola-tions based on a robust industrial dataset We presented Gvt anapproach for automatically detecting classifying and reporting de-sign violations in mobile apps and conducted a wide ranging studythat measured performance utility and industrial applicability ofthis tool Our results indicate that Gvt is effective in practice offersutility for developers and is applicable in industrial contexts

ACKNOWLEDGMENTSThe authors would like to thank Kebing Xie Roozbeh Farahbodand the developers and designers at Huawei for their support

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden

REFERENCES[1] Adobe photoshop httpwwwphotoshopcom[2] Android uiautomator httpdeveloperandroidcomtoolshelpuiautomator

indexhtml[3] Apple app store httpswwwapplecomiosapp-store[4] Fuild-ui httpswwwfluiduicom[5] Google play store httpsplaygooglecomstorehl=en[6] The marketch plugin for sketch httpsgithubcomtudou527marketch[7] Mobile apps What consumers really need and want httpsinfodynatracecom

rscompuwareimagesMobileAppSurveyReportpdf[8] Mockupio httpsmockupioabout[9] Protoio httpsprotoio[10] The sketch design tool httpswwwsketchappcom[11] Sketch extensions httpswwwsketchappcomextensions[12] Why your apprsquos ux is more important than you think httpwwwcodemagcom

Article1401041[13] E Aleacutegroth and R Feldt On the long-term use of visual gui testing in industrial

practice a case study Empirical Software Engineering 22(6)2937ndash2971 Dec 2017[14] E Alegroth Z Gao R Oliveira and A Memon Conceptualization and evaluation

of component-based testing unified with visual gui testing An empirical studyIn 2015 IEEE 8th International Conference on Software Testing Verification andValidation (ICST) pages 1ndash10 April 2015

[15] G Bavota M Linares-Vaacutesquez C Bernal-Caacuterdenas M Di Penta R Oliveto andD Poshyvanyk The impact of api change- and fault-proneness on the user ratingsof android apps Software Engineering IEEE Transactions on 41(4)384ndash407 April2015

[16] J Brooke SUS A quick and dirty usability scale In P W Jordan B WeerdmeesterA Thomas and I L Mclelland editors Usability evaluation in industry Taylorand Francis London 1996

[17] K Charmaz Constructing Grounded Theory SAGE Publications Inc 2006[18] S R Choudhary M R Prasad and A Orso Crosscheck Combining crawling and

differencing to better detect cross-browser incompatibilities in web applicationsIn Proceedings of the 2012 IEEE Fifth International Conference on Software TestingVerification and Validation ICST rsquo12 pages 171ndash180 Washington DC USA 2012IEEE Computer Society

[19] A Ciurumelea A SchaufelbAtildeČAcircijhl S Panichella and H C Gall Analyzingreviews and code of mobile apps for better release planning In 2017 IEEE 24th In-ternational Conference on Software Analysis Evolution and Reengineering (SANER)pages 91ndash102 Feb 2017

[20] A Di Sorbo S Panichella C V Alexandru J Shimagaki C A Visaggio G Can-fora and H C Gall What would users change in my app summarizing appreviews for recommending software changes In Proceedings of the 2016 24thACM SIGSOFT International Symposium on Foundations of Software EngineeringFSE 2016 pages 499ndash510 New York NY USA 2016 ACM

[21] K Holl and F Elberzhager A mobile-specific failure classification and its usageto focus quality assurance In 2014 40th EUROMICRO Conference on SoftwareEngineering and Advanced Applications pages 385ndash388 Aug 2014

[22] G Hu X Yuan Y Tang and J Yang Efficiently effectively detecting mobileapp bugs with appdoctor In Proceedings of the Ninth European Conference onComputer Systems EuroSys rsquo14 pages 181ndash1815 New York NY USA 2014ACM

[23] A Issa J Sillito and V Garousi Visual testing of graphical user interfaces Anexploratory study towards systematic definitions and approaches In 2012 14thIEEE International Symposium on Web Systems Evolution (WSE) pages 11ndash15 Sept2012

[24] N Jones Seven best practices for optimizing mobile testing efforts TechnicalReport G00248240 Gartner

[25] K Kuusinen and T Mikkonen Designing user experience for mobile apps Long-term product owner perspective In 2013 20th Asia-Pacific Software EngineeringConference volume 1 of APSECrsquo13 pages 535ndash540 Dec 2013

[26] V Lelli A Blouin and B Baudry Classifying and qualifying gui defects In 2015IEEE 8th International Conference on Software Testing Verification and Validation(ICST) pages 1ndash10 April 2015

[27] M Linares-Vaacutesquez G Bavota C Bernal-Caacuterdenas M Di Penta R Oliveto andD Poshyvanyk Api change and fault proneness A threat to the success ofandroid apps In Proceedings of the 2013 9th Joint Meeting on Foundations ofSoftware Engineering ESECFSErsquo13 pages 477ndash487 New York NY USA 2013ACM

[28] M Linares-Vaacutesquez G Bavota M D Penta R Oliveto and D Poshyvanyk Howdo API changes trigger Stack Overflow discussions a study on the android SDK

In Proceedings of the 22nd International Conference on Program ComprehensionICPCrsquo14 pages 83ndash94 2014

[29] M Linares-Vaacutesquez G Bavota M Tufano K Moran M Di Penta C VendomeC Bernal-Caacuterdenas and D Poshyvanyk Enabling mutation testing for androidapps In Proceedings of the 2017 11th Joint Meeting on Foundations of SoftwareEngineering ESECFSE 2017 pages 233ndash244 New York NY USA 2017 ACM

[30] S Mahajan A Alameer P McMinn and W G Halfond Automated repair oflayout cross browser issues using search-based techniques In InternationalConference on Software Testing and Analysis ISSTArsquo17 2017

[31] S Mahajan and W G J Halfond Detection and localization of html presentationfailures using computer vision-based techniques In Proceedings of the 8th IEEEInternational Conference on Software Testing Verification and Validation ICSTrsquo15April 2015

[32] S Mahajan B Li P Behnamghader and W G Halfond Using visual symptomsfor debugging presentation failures in web applications In Proceeding of the9th IEEE International Conference on Software Testing Verification and Validation(ICST) ICSTrsquo16 April 2016

[33] T McDonnell B Ray andM Kim An empirical study of api stability and adoptionin the android ecosystem In Proceedings of the 2013 International Conference onSoftware Maintenance ICSMrsquo13 pages 70ndash79 2013

[34] J Meskens K Luyten and K Coninx Plug-and-design Embracing mobiledevices as part of the design environment In Proceedings of the 1st ACM SIGCHISymposium on Engineering Interactive Computing Systems EICS rsquo09 pages 149ndash154 New York NY USA 2009 ACM

[35] K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk Gvt onlineappendix httpwwwandroid-dev-toolscomgvt

[36] P Morville User experience design httpsemanticstudioscomuser_experience_design

[37] B Myers Challenges of hci design and implementation interactions 1(1)73ndash83Jan 1994

[38] B N Nguyen B Robbins I Banerjee and A Memon Guitar An innovativetool for automated testing of gui-driven software Automated Software Engg21(1)65ndash105 Mar 2014

[39] T A Nguyen and C Csallner Reverse engineering mobile application userinterfaces with REMAUI In Proceedings of the 2015 30th IEEEACM Interna-tional Conference on Automated Software Engineering ASErsquo15 pages 248ndash259Washington DC USA 2015 IEEE Computer Society

[40] F PalombaM Linares-Vaacutesquez G Bavota R Oliveto M D Penta D Poshyvanykand A D Lucia User reviews matter tracking crowdsourced reviews to supportevolution of successful apps In 2015 IEEE International Conference on SoftwareMaintenance and Evolution (ICSME) pages 291ndash300 Sept 2015

[41] F Palomba P Salza A Ciurumelea S Panichella H Gall F Ferrucci andA De Lucia Recommending and localizing change requests for mobile appsbased on user reviews In Proceedings of the 39th International Conference onSoftware Engineering ICSE rsquo17 pages 106ndash117 Piscataway NJ USA 2017 IEEEPress

[42] S Roy Choudhary M R Prasad and A Orso X-pert Accurate identification ofcross-browser issues in web applications In Proceedings of the 2013 InternationalConference on Software Engineering ICSE rsquo13 pages 702ndash711 Piscataway NJUSA 2013 IEEE Press

[43] S Roy Choudhary H Versee and A Orso Webdiff Automated identification ofcross-browser issues in web applications In Proceedings of the 2010 IEEE Interna-tional Conference on Software Maintenance ICSM rsquo10 pages 1ndash10 WashingtonDC USA 2010 IEEE Computer Society

[44] T Silva da Silva A Martin F Maurer and M Silveira User-centered design andagile methods A systematic review In Proceedings of the 2011 Agile ConferenceAGILE rsquo11 pages 77ndash86 Washington DC USA 2011 IEEE Computer Society

[45] K-J Stol P Ralph and B Fitzgerald Grounded theory in software engineeringresearch A critical review and guidelines In Proceedings of the 38th InternationalConference on Software Engineering ICSE rsquo16 pages 120ndash131 New York NY USA2016 ACM

[46] A B Tucker Computer Science Handbook Second Edition Chapman amp HallCRC2004

[47] Q Xie M Grechanik C Fu and C Cumby Guide A gui differentiator In 2009IEEE International Conference on Software Maintenance ICSMrsquo09

[48] H Yee S Pattanaik and D P Greenberg Spatiotemporal sensitivity and visualattention for efficient rendering of dynamic environments ACM Trans Graph20(1)39ndash65 Jan 2001

[49] C Zeidler C Lutteroth W Stuerzlinger and G Weber Evaluating Direct Manip-ulation Operations for Constraint-Based Layout pages 513ndash529 Springer BerlinHeidelberg Berlin Heidelberg 2013

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden

CPDP

R

000 025 050 075 100

Figure 5 Study 2 - Developer CP DP and R

I found these reports unnecessarily complex

I found these reports very cumbersome to read

I think that I would like to use these reports frequently for Identifying Presentation Issues

I thought these reports were very useful for accurately identifying Presentation Errors

These reports are easy to readunderstand

SD D N A SA

Figure 6 Study 2 - UXQuestion Responses SD=Strongly Dis-agree D=Disagree N=Neutral A=Agree SA=Strongly Agree

the average R reaching 965 This illustrates that Gvt is capableof detecting seeded faults designed to emulate both the type anddistribution of DVs encountered in industrial settings While Gvtachieved at least 85 precision for each type of seeded DV it per-formed worse on some types of violations compared to others Forinstance Gvt saw its lowest precision values for the Font-Style andFont-Color violations typically due to the fact that the magnitudeof perturbation for the color or font type was not large enough tosurpass the Color or Image Difference Thresholds (CT amp IDT ) Gvttook 368 mins to process and generate reports for the set of 100screens with injected DVs or 22 sec per screen pair This executioncost was generally acceptable by our industrial collaborators

62 Study 2 Results GVT UtilityThe DP CP and R results representing the Android developersability to correctly detect and classify DVs is shown in Figure 5as box-plots across all 10 participants Here we found CP=DP aswhen a user misclassified violations they also did not detect themAs this figure shows the Android developers generally performedmuch worse compared to Gvt achieving an average CP of underasymp 60 and an average R of asymp 50 The sources of this performanceloss for the study participants compared to Gvt was fourfold (i)participants tended to report minor acceptable differences in fontsacross the examples (despite the instructions clearly stating notto report such violations) (ii) users tended to attribute more thanone DV to a single component specifically for font style and fontcolor violations despite instructions to report only one (iii) userstended to misclassify DVs based on the provided categories (egclassifying a layout DV for a Text GC as an incorrect text DV ) and(iv) participants missed reporting many of the injected DVs lead-ing to the low recall numbers These results indicate that at thevery least developers can struggle to both detect and classify DVsbetween mock-up and implementation screen pairs signaling theneed for an automated system to check for DVs before implementedapps are sent to a UIUX team for auditing This result confirmsthe notion that developers may not be as sensitive to small DVsin the GUI as the designers who created the GUI specificationsFurthermore this finding is notable because as part of the iterativeprocess of resolving design violations designers must communicateto developers DVs and developers must recognize and understandthese DVs in order to properly resolve them This process is oftencomplicated due to ambiguous descriptions of DVs from designers

The GVT allowed for better transfer of the design from mockminusups to the implementation of the app

The GVT has helped you to reduce the time required for verifying design violations

The GVT is able to accurately report existing design violations in productionminusquality applications

The GVT tool helped my team (designimplementation) communicate with other teams (implementationdesign)

regarding GUI design violations

Using the GUIminusVerification tool (GVT) helped to improve the quality of mobile

applications produced by industrial partner

SD D N A SA

Figure 7 Study 3 - Applicability Questions SD=Strongly Dis-agree D=Disagree N=Neutral A=Agree SA=Strongly Agree

to developers or developers disagreeing with designers over the ex-istence or type of a DV In contrast to this fragmented process Gvtprovides clear unambiguous reports that facilitate communicationbetween designers and developers

Figure 6 illustrates the responses to the likert based UX ques-tions and the results are quite encouraging In general participantsfound that the reports from Gvt were easy to read useful for iden-tifying DVs and indicated that they would like to use the reports foridentifying DVs Participants also indicated that the reports werenot unnecessarily complex or difficult to read We asked the partic-ipants about their preferences for the Gvt reports as well askingabout the most and least useful information in the reports Everysingle participant indicated that the highlighted annotations on thescreenshots in the report were the most useful element Whereasmost users tended to dislike the PID output included at the bottomof the report citing this information as difficult to comprehend

63 Study 3 Results Industrial ApplicabilityThe results for the applicability questions asked to 20 designersand developers who use Gvt in their daily activities is shown inFigure 7 A positive outcome for each of these statements correlatesto responses indicating that developers ldquoagreerdquo or ldquostrongly agreerdquoThe results of this study indicate a weak agreement of developersfor these statements indicating that while Gvt is generally appli-cable there are some drawbacks that prevented developers anddesigners from giving the tool unequivocal support We explorethese drawbacks by conducting semi-structured interviews

In conducting the interviews one of the authors asked the ques-tions presented in Figure 7 to 3 managers (2 from UIUX teams and1 from a Front-End development team) When asked whether Gvtcontributed to an increased quality of mobile applications at thecompany all three managers tended to agree that this was the caseFor instance one of the design managers stated ldquoCertainly yes Thetool is the industryrsquos first and the other designer manager addedldquoWhen the page is more complicated the tool is more helpful

When asked about the overall performance and accuracy of thetool in detecting DVs the manager from the implementation teamadmitted that the current detection performance of the tool is goodbut suggested that dynamic detection of some components mayimprove it stating ldquo[DVs ] can be detected pretty well [but the toolis] not very flexible For example a switch component in the designis open but the switch is off in the implementation He suggestedthat properly handling cases such as this would make the toolmore useful from a developers perspective One of the design teammanagers held a similar view stating that ldquoCurrently most errorsare layout errors so tool is accurate Static components are basicallydetected [but] maybe the next extension should focus on dynamic

ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk

components While the current version of the Gvt allows for theexclusion of regions with dynamic components it is clear that bothdesign and development teams would appreciate proper detectionof DVs for dynamic components Additionally two of the managerscommented on the ldquorigidityrdquo of the Gvtrsquos current interface andexplained that a more streamlined UI would help improve its utility

When asked about whether Gvt improved communication be-tween the design and development teams the development teammanager felt that while the tool has not improved communica-tion yet it did have the potential to do so ldquoAt present there is no[improvement] but certainly there is the potential possibility Thedesign managers generally stated that the tool has helped withcommunication particularly in clarifying subtle DVs that may havecaused arguments between teams in the past ldquoIf you consider thetime savings on discussion and arguments between the two teamsthis tool saves us a lot of time Another designer indicated that thetool is helpful at describing DVs to developers who may not be ableto recognize them with the naked eye ldquoWe found that the tool canindeed detect something that the naked eye cannot While there arecertainly further refinements that can be made to Gvt it is clearthat the tool has begun to have a positive impact of the developmentof mobile apps and as the tool evolves within the company shouldallow for continued improvements in quality and time saved

7 LIMITATIONS amp THREATS TO VALIDITYLimitations While we have illustrated thatGvt is applicable in anindustrial setting the tool is not without its limitations Currentlythe tool imposes lightweight restrictions on designers creatingSketch mock-ups chief among these being the requirement thatbounding boxes of components do not overlap Currently Gvt willtry to resolve such cases during the GUI-Comprehension stage usingan Intersection over union (IOU) metricInternal Validity While deriving the taxonomy of DVs mistakesin classification arising from subjectiveness may have introducedunexpected coding Tomitigate this threat we followed a setmethod-ology merged coding results and performed conflict resolutionConstruct Validity In our initial study (Sec 3) a threat to con-struct validity arises in the form of themanner in which coders wereexposed to presentation failures To mitigate this threat designersfrom our industrial partner manually annotated the screen pairsin order to clearly illustrate the affected GCs on the screen In ourevaluation of Gvt threats arise from our method of DV injectionusing the synthetic fault injection tool However we designed thistool to inject faults based upon both the type and distribution offaults from our DV taxonomy to mitigate this threatExternal Validity In our initial study related to the DV taxonomywe utilized a dataset from a single (albeit large) company withexamples across several different applications and screens There isthe potential that this may not generalize to other industrial mobileapplication development environments and platforms or mobile appdevelopment in general However given the relatively consistentdesign paradigms of mobile apps we expect the categories andthe sub-categories within the taxonomy to hold although it ispossible that the distribution across these categories may varyacross application development for different domains In Study 3we surveyed employees at a single (though large) company andfindings may differ in similar studies at other companies

8 RELATEDWORKWeb Presentation Failures The work most closely related to ourapproach are approaches that aim at detecting classifying and fixingpresentation failures in web applications [30ndash32 42] In comparisonto these approaches Gvt also performs detection and localizationof presentation failures but is the first to do so for mobile appsIn addition to the engineering challenges associated with buildingan approach to detect presentation failures in the mobile domain(eg collection and processing of GUI-related data) Gvt is the firstapproach to leverage metadata from software mock-up artifacts(eg Marketch) to perform GC matching based upon the spatialinformation collected from both mock-ups and dynamic applicationscreens allowing for precise detection of the different types of DVsdelineated in our industrial DV taxonomy Gvt is also the first toapply the processes of CQ CH analysis and B-PID toward detectingdifferences in the content and color of icons and images displayedin mobile apps Gvt also explicitly identifies and reports differentfaulty properties (such as errors in component location or text)Cross Browser Testing Approaches for XBT (or cross browsertesting) by Roy Choudhry et al [18 42 43] examine and automati-cally report differences in web pages rendered in multiple browsersThese approaches are currently not directly applicable to mock-updriven development for mobile appsVisualGUI Testing A concept known as Visual GUI Testing (VGT)aims to test certain visual aspects of a software applicationrsquos GUIas well as the underlying functional properties To accomplish thisvisual GUI testing usually executes actions on a target applicationsin order to exercise app functionality [13 14 23 38] In contrastto these approaches Gvt is designed to apply to mobile-specificDVs is tailored for the mock-up driven development practice andis aimed only at verifying visual properties of a mobile apprsquos GUIOther Approaches There are other approaches and techniquesthat related to identifying problems or differences with GUIs ofmobile apps Xie et al introduced GUIDE [47] a tool for GUI differ-encing between successive releases of GUIs for an app by matchingcomponents between GUI-hierarchies Gvt utilizes a matching pro-cedure for leaf node components as direct tree comparisons are notpossible in the context of mock-up driven development There hasalso been both commercial and academic work related to graphi-cal software built specifically for creating high-fidelity mobile appmock-ups or mockups that encode information for automated cre-ation of code for a target platform [4 8 9 34] However such toolstend to either impose too many restrictions on designers or do notallow for direct creation of code thus DVs still persist in practice9 CONCLUSION amp FUTUREWORKIn this paper we have formalized the problem of detecting designviolations in mobile apps and derived a taxonomy of design viola-tions based on a robust industrial dataset We presented Gvt anapproach for automatically detecting classifying and reporting de-sign violations in mobile apps and conducted a wide ranging studythat measured performance utility and industrial applicability ofthis tool Our results indicate that Gvt is effective in practice offersutility for developers and is applicable in industrial contexts

ACKNOWLEDGMENTSThe authors would like to thank Kebing Xie Roozbeh Farahbodand the developers and designers at Huawei for their support

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden

REFERENCES[1] Adobe photoshop httpwwwphotoshopcom[2] Android uiautomator httpdeveloperandroidcomtoolshelpuiautomator

indexhtml[3] Apple app store httpswwwapplecomiosapp-store[4] Fuild-ui httpswwwfluiduicom[5] Google play store httpsplaygooglecomstorehl=en[6] The marketch plugin for sketch httpsgithubcomtudou527marketch[7] Mobile apps What consumers really need and want httpsinfodynatracecom

rscompuwareimagesMobileAppSurveyReportpdf[8] Mockupio httpsmockupioabout[9] Protoio httpsprotoio[10] The sketch design tool httpswwwsketchappcom[11] Sketch extensions httpswwwsketchappcomextensions[12] Why your apprsquos ux is more important than you think httpwwwcodemagcom

Article1401041[13] E Aleacutegroth and R Feldt On the long-term use of visual gui testing in industrial

practice a case study Empirical Software Engineering 22(6)2937ndash2971 Dec 2017[14] E Alegroth Z Gao R Oliveira and A Memon Conceptualization and evaluation

of component-based testing unified with visual gui testing An empirical studyIn 2015 IEEE 8th International Conference on Software Testing Verification andValidation (ICST) pages 1ndash10 April 2015

[15] G Bavota M Linares-Vaacutesquez C Bernal-Caacuterdenas M Di Penta R Oliveto andD Poshyvanyk The impact of api change- and fault-proneness on the user ratingsof android apps Software Engineering IEEE Transactions on 41(4)384ndash407 April2015

[16] J Brooke SUS A quick and dirty usability scale In P W Jordan B WeerdmeesterA Thomas and I L Mclelland editors Usability evaluation in industry Taylorand Francis London 1996

[17] K Charmaz Constructing Grounded Theory SAGE Publications Inc 2006[18] S R Choudhary M R Prasad and A Orso Crosscheck Combining crawling and

differencing to better detect cross-browser incompatibilities in web applicationsIn Proceedings of the 2012 IEEE Fifth International Conference on Software TestingVerification and Validation ICST rsquo12 pages 171ndash180 Washington DC USA 2012IEEE Computer Society

[19] A Ciurumelea A SchaufelbAtildeČAcircijhl S Panichella and H C Gall Analyzingreviews and code of mobile apps for better release planning In 2017 IEEE 24th In-ternational Conference on Software Analysis Evolution and Reengineering (SANER)pages 91ndash102 Feb 2017

[20] A Di Sorbo S Panichella C V Alexandru J Shimagaki C A Visaggio G Can-fora and H C Gall What would users change in my app summarizing appreviews for recommending software changes In Proceedings of the 2016 24thACM SIGSOFT International Symposium on Foundations of Software EngineeringFSE 2016 pages 499ndash510 New York NY USA 2016 ACM

[21] K Holl and F Elberzhager A mobile-specific failure classification and its usageto focus quality assurance In 2014 40th EUROMICRO Conference on SoftwareEngineering and Advanced Applications pages 385ndash388 Aug 2014

[22] G Hu X Yuan Y Tang and J Yang Efficiently effectively detecting mobileapp bugs with appdoctor In Proceedings of the Ninth European Conference onComputer Systems EuroSys rsquo14 pages 181ndash1815 New York NY USA 2014ACM

[23] A Issa J Sillito and V Garousi Visual testing of graphical user interfaces Anexploratory study towards systematic definitions and approaches In 2012 14thIEEE International Symposium on Web Systems Evolution (WSE) pages 11ndash15 Sept2012

[24] N Jones Seven best practices for optimizing mobile testing efforts TechnicalReport G00248240 Gartner

[25] K Kuusinen and T Mikkonen Designing user experience for mobile apps Long-term product owner perspective In 2013 20th Asia-Pacific Software EngineeringConference volume 1 of APSECrsquo13 pages 535ndash540 Dec 2013

[26] V Lelli A Blouin and B Baudry Classifying and qualifying gui defects In 2015IEEE 8th International Conference on Software Testing Verification and Validation(ICST) pages 1ndash10 April 2015

[27] M Linares-Vaacutesquez G Bavota C Bernal-Caacuterdenas M Di Penta R Oliveto andD Poshyvanyk Api change and fault proneness A threat to the success ofandroid apps In Proceedings of the 2013 9th Joint Meeting on Foundations ofSoftware Engineering ESECFSErsquo13 pages 477ndash487 New York NY USA 2013ACM

[28] M Linares-Vaacutesquez G Bavota M D Penta R Oliveto and D Poshyvanyk Howdo API changes trigger Stack Overflow discussions a study on the android SDK

In Proceedings of the 22nd International Conference on Program ComprehensionICPCrsquo14 pages 83ndash94 2014

[29] M Linares-Vaacutesquez G Bavota M Tufano K Moran M Di Penta C VendomeC Bernal-Caacuterdenas and D Poshyvanyk Enabling mutation testing for androidapps In Proceedings of the 2017 11th Joint Meeting on Foundations of SoftwareEngineering ESECFSE 2017 pages 233ndash244 New York NY USA 2017 ACM

[30] S Mahajan A Alameer P McMinn and W G Halfond Automated repair oflayout cross browser issues using search-based techniques In InternationalConference on Software Testing and Analysis ISSTArsquo17 2017

[31] S Mahajan and W G J Halfond Detection and localization of html presentationfailures using computer vision-based techniques In Proceedings of the 8th IEEEInternational Conference on Software Testing Verification and Validation ICSTrsquo15April 2015

[32] S Mahajan B Li P Behnamghader and W G Halfond Using visual symptomsfor debugging presentation failures in web applications In Proceeding of the9th IEEE International Conference on Software Testing Verification and Validation(ICST) ICSTrsquo16 April 2016

[33] T McDonnell B Ray andM Kim An empirical study of api stability and adoptionin the android ecosystem In Proceedings of the 2013 International Conference onSoftware Maintenance ICSMrsquo13 pages 70ndash79 2013

[34] J Meskens K Luyten and K Coninx Plug-and-design Embracing mobiledevices as part of the design environment In Proceedings of the 1st ACM SIGCHISymposium on Engineering Interactive Computing Systems EICS rsquo09 pages 149ndash154 New York NY USA 2009 ACM

[35] K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk Gvt onlineappendix httpwwwandroid-dev-toolscomgvt

[36] P Morville User experience design httpsemanticstudioscomuser_experience_design

[37] B Myers Challenges of hci design and implementation interactions 1(1)73ndash83Jan 1994

[38] B N Nguyen B Robbins I Banerjee and A Memon Guitar An innovativetool for automated testing of gui-driven software Automated Software Engg21(1)65ndash105 Mar 2014

[39] T A Nguyen and C Csallner Reverse engineering mobile application userinterfaces with REMAUI In Proceedings of the 2015 30th IEEEACM Interna-tional Conference on Automated Software Engineering ASErsquo15 pages 248ndash259Washington DC USA 2015 IEEE Computer Society

[40] F PalombaM Linares-Vaacutesquez G Bavota R Oliveto M D Penta D Poshyvanykand A D Lucia User reviews matter tracking crowdsourced reviews to supportevolution of successful apps In 2015 IEEE International Conference on SoftwareMaintenance and Evolution (ICSME) pages 291ndash300 Sept 2015

[41] F Palomba P Salza A Ciurumelea S Panichella H Gall F Ferrucci andA De Lucia Recommending and localizing change requests for mobile appsbased on user reviews In Proceedings of the 39th International Conference onSoftware Engineering ICSE rsquo17 pages 106ndash117 Piscataway NJ USA 2017 IEEEPress

[42] S Roy Choudhary M R Prasad and A Orso X-pert Accurate identification ofcross-browser issues in web applications In Proceedings of the 2013 InternationalConference on Software Engineering ICSE rsquo13 pages 702ndash711 Piscataway NJUSA 2013 IEEE Press

[43] S Roy Choudhary H Versee and A Orso Webdiff Automated identification ofcross-browser issues in web applications In Proceedings of the 2010 IEEE Interna-tional Conference on Software Maintenance ICSM rsquo10 pages 1ndash10 WashingtonDC USA 2010 IEEE Computer Society

[44] T Silva da Silva A Martin F Maurer and M Silveira User-centered design andagile methods A systematic review In Proceedings of the 2011 Agile ConferenceAGILE rsquo11 pages 77ndash86 Washington DC USA 2011 IEEE Computer Society

[45] K-J Stol P Ralph and B Fitzgerald Grounded theory in software engineeringresearch A critical review and guidelines In Proceedings of the 38th InternationalConference on Software Engineering ICSE rsquo16 pages 120ndash131 New York NY USA2016 ACM

[46] A B Tucker Computer Science Handbook Second Edition Chapman amp HallCRC2004

[47] Q Xie M Grechanik C Fu and C Cumby Guide A gui differentiator In 2009IEEE International Conference on Software Maintenance ICSMrsquo09

[48] H Yee S Pattanaik and D P Greenberg Spatiotemporal sensitivity and visualattention for efficient rendering of dynamic environments ACM Trans Graph20(1)39ndash65 Jan 2001

[49] C Zeidler C Lutteroth W Stuerzlinger and G Weber Evaluating Direct Manip-ulation Operations for Constraint-Based Layout pages 513ndash529 Springer BerlinHeidelberg Berlin Heidelberg 2013

ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk

components While the current version of the Gvt allows for theexclusion of regions with dynamic components it is clear that bothdesign and development teams would appreciate proper detectionof DVs for dynamic components Additionally two of the managerscommented on the ldquorigidityrdquo of the Gvtrsquos current interface andexplained that a more streamlined UI would help improve its utility

When asked about whether Gvt improved communication be-tween the design and development teams the development teammanager felt that while the tool has not improved communica-tion yet it did have the potential to do so ldquoAt present there is no[improvement] but certainly there is the potential possibility Thedesign managers generally stated that the tool has helped withcommunication particularly in clarifying subtle DVs that may havecaused arguments between teams in the past ldquoIf you consider thetime savings on discussion and arguments between the two teamsthis tool saves us a lot of time Another designer indicated that thetool is helpful at describing DVs to developers who may not be ableto recognize them with the naked eye ldquoWe found that the tool canindeed detect something that the naked eye cannot While there arecertainly further refinements that can be made to Gvt it is clearthat the tool has begun to have a positive impact of the developmentof mobile apps and as the tool evolves within the company shouldallow for continued improvements in quality and time saved

7 LIMITATIONS amp THREATS TO VALIDITYLimitations While we have illustrated thatGvt is applicable in anindustrial setting the tool is not without its limitations Currentlythe tool imposes lightweight restrictions on designers creatingSketch mock-ups chief among these being the requirement thatbounding boxes of components do not overlap Currently Gvt willtry to resolve such cases during the GUI-Comprehension stage usingan Intersection over union (IOU) metricInternal Validity While deriving the taxonomy of DVs mistakesin classification arising from subjectiveness may have introducedunexpected coding Tomitigate this threat we followed a setmethod-ology merged coding results and performed conflict resolutionConstruct Validity In our initial study (Sec 3) a threat to con-struct validity arises in the form of themanner in which coders wereexposed to presentation failures To mitigate this threat designersfrom our industrial partner manually annotated the screen pairsin order to clearly illustrate the affected GCs on the screen In ourevaluation of Gvt threats arise from our method of DV injectionusing the synthetic fault injection tool However we designed thistool to inject faults based upon both the type and distribution offaults from our DV taxonomy to mitigate this threatExternal Validity In our initial study related to the DV taxonomywe utilized a dataset from a single (albeit large) company withexamples across several different applications and screens There isthe potential that this may not generalize to other industrial mobileapplication development environments and platforms or mobile appdevelopment in general However given the relatively consistentdesign paradigms of mobile apps we expect the categories andthe sub-categories within the taxonomy to hold although it ispossible that the distribution across these categories may varyacross application development for different domains In Study 3we surveyed employees at a single (though large) company andfindings may differ in similar studies at other companies

8 RELATEDWORKWeb Presentation Failures The work most closely related to ourapproach are approaches that aim at detecting classifying and fixingpresentation failures in web applications [30ndash32 42] In comparisonto these approaches Gvt also performs detection and localizationof presentation failures but is the first to do so for mobile appsIn addition to the engineering challenges associated with buildingan approach to detect presentation failures in the mobile domain(eg collection and processing of GUI-related data) Gvt is the firstapproach to leverage metadata from software mock-up artifacts(eg Marketch) to perform GC matching based upon the spatialinformation collected from both mock-ups and dynamic applicationscreens allowing for precise detection of the different types of DVsdelineated in our industrial DV taxonomy Gvt is also the first toapply the processes of CQ CH analysis and B-PID toward detectingdifferences in the content and color of icons and images displayedin mobile apps Gvt also explicitly identifies and reports differentfaulty properties (such as errors in component location or text)Cross Browser Testing Approaches for XBT (or cross browsertesting) by Roy Choudhry et al [18 42 43] examine and automati-cally report differences in web pages rendered in multiple browsersThese approaches are currently not directly applicable to mock-updriven development for mobile appsVisualGUI Testing A concept known as Visual GUI Testing (VGT)aims to test certain visual aspects of a software applicationrsquos GUIas well as the underlying functional properties To accomplish thisvisual GUI testing usually executes actions on a target applicationsin order to exercise app functionality [13 14 23 38] In contrastto these approaches Gvt is designed to apply to mobile-specificDVs is tailored for the mock-up driven development practice andis aimed only at verifying visual properties of a mobile apprsquos GUIOther Approaches There are other approaches and techniquesthat related to identifying problems or differences with GUIs ofmobile apps Xie et al introduced GUIDE [47] a tool for GUI differ-encing between successive releases of GUIs for an app by matchingcomponents between GUI-hierarchies Gvt utilizes a matching pro-cedure for leaf node components as direct tree comparisons are notpossible in the context of mock-up driven development There hasalso been both commercial and academic work related to graphi-cal software built specifically for creating high-fidelity mobile appmock-ups or mockups that encode information for automated cre-ation of code for a target platform [4 8 9 34] However such toolstend to either impose too many restrictions on designers or do notallow for direct creation of code thus DVs still persist in practice9 CONCLUSION amp FUTUREWORKIn this paper we have formalized the problem of detecting designviolations in mobile apps and derived a taxonomy of design viola-tions based on a robust industrial dataset We presented Gvt anapproach for automatically detecting classifying and reporting de-sign violations in mobile apps and conducted a wide ranging studythat measured performance utility and industrial applicability ofthis tool Our results indicate that Gvt is effective in practice offersutility for developers and is applicable in industrial contexts

ACKNOWLEDGMENTSThe authors would like to thank Kebing Xie Roozbeh Farahbodand the developers and designers at Huawei for their support

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden

REFERENCES[1] Adobe photoshop httpwwwphotoshopcom[2] Android uiautomator httpdeveloperandroidcomtoolshelpuiautomator

indexhtml[3] Apple app store httpswwwapplecomiosapp-store[4] Fuild-ui httpswwwfluiduicom[5] Google play store httpsplaygooglecomstorehl=en[6] The marketch plugin for sketch httpsgithubcomtudou527marketch[7] Mobile apps What consumers really need and want httpsinfodynatracecom

rscompuwareimagesMobileAppSurveyReportpdf[8] Mockupio httpsmockupioabout[9] Protoio httpsprotoio[10] The sketch design tool httpswwwsketchappcom[11] Sketch extensions httpswwwsketchappcomextensions[12] Why your apprsquos ux is more important than you think httpwwwcodemagcom

Article1401041[13] E Aleacutegroth and R Feldt On the long-term use of visual gui testing in industrial

practice a case study Empirical Software Engineering 22(6)2937ndash2971 Dec 2017[14] E Alegroth Z Gao R Oliveira and A Memon Conceptualization and evaluation

of component-based testing unified with visual gui testing An empirical studyIn 2015 IEEE 8th International Conference on Software Testing Verification andValidation (ICST) pages 1ndash10 April 2015

[15] G Bavota M Linares-Vaacutesquez C Bernal-Caacuterdenas M Di Penta R Oliveto andD Poshyvanyk The impact of api change- and fault-proneness on the user ratingsof android apps Software Engineering IEEE Transactions on 41(4)384ndash407 April2015

[16] J Brooke SUS A quick and dirty usability scale In P W Jordan B WeerdmeesterA Thomas and I L Mclelland editors Usability evaluation in industry Taylorand Francis London 1996

[17] K Charmaz Constructing Grounded Theory SAGE Publications Inc 2006[18] S R Choudhary M R Prasad and A Orso Crosscheck Combining crawling and

differencing to better detect cross-browser incompatibilities in web applicationsIn Proceedings of the 2012 IEEE Fifth International Conference on Software TestingVerification and Validation ICST rsquo12 pages 171ndash180 Washington DC USA 2012IEEE Computer Society

[19] A Ciurumelea A SchaufelbAtildeČAcircijhl S Panichella and H C Gall Analyzingreviews and code of mobile apps for better release planning In 2017 IEEE 24th In-ternational Conference on Software Analysis Evolution and Reengineering (SANER)pages 91ndash102 Feb 2017

[20] A Di Sorbo S Panichella C V Alexandru J Shimagaki C A Visaggio G Can-fora and H C Gall What would users change in my app summarizing appreviews for recommending software changes In Proceedings of the 2016 24thACM SIGSOFT International Symposium on Foundations of Software EngineeringFSE 2016 pages 499ndash510 New York NY USA 2016 ACM

[21] K Holl and F Elberzhager A mobile-specific failure classification and its usageto focus quality assurance In 2014 40th EUROMICRO Conference on SoftwareEngineering and Advanced Applications pages 385ndash388 Aug 2014

[22] G Hu X Yuan Y Tang and J Yang Efficiently effectively detecting mobileapp bugs with appdoctor In Proceedings of the Ninth European Conference onComputer Systems EuroSys rsquo14 pages 181ndash1815 New York NY USA 2014ACM

[23] A Issa J Sillito and V Garousi Visual testing of graphical user interfaces Anexploratory study towards systematic definitions and approaches In 2012 14thIEEE International Symposium on Web Systems Evolution (WSE) pages 11ndash15 Sept2012

[24] N Jones Seven best practices for optimizing mobile testing efforts TechnicalReport G00248240 Gartner

[25] K Kuusinen and T Mikkonen Designing user experience for mobile apps Long-term product owner perspective In 2013 20th Asia-Pacific Software EngineeringConference volume 1 of APSECrsquo13 pages 535ndash540 Dec 2013

[26] V Lelli A Blouin and B Baudry Classifying and qualifying gui defects In 2015IEEE 8th International Conference on Software Testing Verification and Validation(ICST) pages 1ndash10 April 2015

[27] M Linares-Vaacutesquez G Bavota C Bernal-Caacuterdenas M Di Penta R Oliveto andD Poshyvanyk Api change and fault proneness A threat to the success ofandroid apps In Proceedings of the 2013 9th Joint Meeting on Foundations ofSoftware Engineering ESECFSErsquo13 pages 477ndash487 New York NY USA 2013ACM

[28] M Linares-Vaacutesquez G Bavota M D Penta R Oliveto and D Poshyvanyk Howdo API changes trigger Stack Overflow discussions a study on the android SDK

In Proceedings of the 22nd International Conference on Program ComprehensionICPCrsquo14 pages 83ndash94 2014

[29] M Linares-Vaacutesquez G Bavota M Tufano K Moran M Di Penta C VendomeC Bernal-Caacuterdenas and D Poshyvanyk Enabling mutation testing for androidapps In Proceedings of the 2017 11th Joint Meeting on Foundations of SoftwareEngineering ESECFSE 2017 pages 233ndash244 New York NY USA 2017 ACM

[30] S Mahajan A Alameer P McMinn and W G Halfond Automated repair oflayout cross browser issues using search-based techniques In InternationalConference on Software Testing and Analysis ISSTArsquo17 2017

[31] S Mahajan and W G J Halfond Detection and localization of html presentationfailures using computer vision-based techniques In Proceedings of the 8th IEEEInternational Conference on Software Testing Verification and Validation ICSTrsquo15April 2015

[32] S Mahajan B Li P Behnamghader and W G Halfond Using visual symptomsfor debugging presentation failures in web applications In Proceeding of the9th IEEE International Conference on Software Testing Verification and Validation(ICST) ICSTrsquo16 April 2016

[33] T McDonnell B Ray andM Kim An empirical study of api stability and adoptionin the android ecosystem In Proceedings of the 2013 International Conference onSoftware Maintenance ICSMrsquo13 pages 70ndash79 2013

[34] J Meskens K Luyten and K Coninx Plug-and-design Embracing mobiledevices as part of the design environment In Proceedings of the 1st ACM SIGCHISymposium on Engineering Interactive Computing Systems EICS rsquo09 pages 149ndash154 New York NY USA 2009 ACM

[35] K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk Gvt onlineappendix httpwwwandroid-dev-toolscomgvt

[36] P Morville User experience design httpsemanticstudioscomuser_experience_design

[37] B Myers Challenges of hci design and implementation interactions 1(1)73ndash83Jan 1994

[38] B N Nguyen B Robbins I Banerjee and A Memon Guitar An innovativetool for automated testing of gui-driven software Automated Software Engg21(1)65ndash105 Mar 2014

[39] T A Nguyen and C Csallner Reverse engineering mobile application userinterfaces with REMAUI In Proceedings of the 2015 30th IEEEACM Interna-tional Conference on Automated Software Engineering ASErsquo15 pages 248ndash259Washington DC USA 2015 IEEE Computer Society

[40] F PalombaM Linares-Vaacutesquez G Bavota R Oliveto M D Penta D Poshyvanykand A D Lucia User reviews matter tracking crowdsourced reviews to supportevolution of successful apps In 2015 IEEE International Conference on SoftwareMaintenance and Evolution (ICSME) pages 291ndash300 Sept 2015

[41] F Palomba P Salza A Ciurumelea S Panichella H Gall F Ferrucci andA De Lucia Recommending and localizing change requests for mobile appsbased on user reviews In Proceedings of the 39th International Conference onSoftware Engineering ICSE rsquo17 pages 106ndash117 Piscataway NJ USA 2017 IEEEPress

[42] S Roy Choudhary M R Prasad and A Orso X-pert Accurate identification ofcross-browser issues in web applications In Proceedings of the 2013 InternationalConference on Software Engineering ICSE rsquo13 pages 702ndash711 Piscataway NJUSA 2013 IEEE Press

[43] S Roy Choudhary H Versee and A Orso Webdiff Automated identification ofcross-browser issues in web applications In Proceedings of the 2010 IEEE Interna-tional Conference on Software Maintenance ICSM rsquo10 pages 1ndash10 WashingtonDC USA 2010 IEEE Computer Society

[44] T Silva da Silva A Martin F Maurer and M Silveira User-centered design andagile methods A systematic review In Proceedings of the 2011 Agile ConferenceAGILE rsquo11 pages 77ndash86 Washington DC USA 2011 IEEE Computer Society

[45] K-J Stol P Ralph and B Fitzgerald Grounded theory in software engineeringresearch A critical review and guidelines In Proceedings of the 38th InternationalConference on Software Engineering ICSE rsquo16 pages 120ndash131 New York NY USA2016 ACM

[46] A B Tucker Computer Science Handbook Second Edition Chapman amp HallCRC2004

[47] Q Xie M Grechanik C Fu and C Cumby Guide A gui differentiator In 2009IEEE International Conference on Software Maintenance ICSMrsquo09

[48] H Yee S Pattanaik and D P Greenberg Spatiotemporal sensitivity and visualattention for efficient rendering of dynamic environments ACM Trans Graph20(1)39ndash65 Jan 2001

[49] C Zeidler C Lutteroth W Stuerzlinger and G Weber Evaluating Direct Manip-ulation Operations for Constraint-Based Layout pages 513ndash529 Springer BerlinHeidelberg Berlin Heidelberg 2013

Automated Reporting of GUI Design Violations for Mobile Apps ICSE rsquo18 May 27-June 3 2018 Gothenburg Sweden

REFERENCES[1] Adobe photoshop httpwwwphotoshopcom[2] Android uiautomator httpdeveloperandroidcomtoolshelpuiautomator

indexhtml[3] Apple app store httpswwwapplecomiosapp-store[4] Fuild-ui httpswwwfluiduicom[5] Google play store httpsplaygooglecomstorehl=en[6] The marketch plugin for sketch httpsgithubcomtudou527marketch[7] Mobile apps What consumers really need and want httpsinfodynatracecom

rscompuwareimagesMobileAppSurveyReportpdf[8] Mockupio httpsmockupioabout[9] Protoio httpsprotoio[10] The sketch design tool httpswwwsketchappcom[11] Sketch extensions httpswwwsketchappcomextensions[12] Why your apprsquos ux is more important than you think httpwwwcodemagcom

Article1401041[13] E Aleacutegroth and R Feldt On the long-term use of visual gui testing in industrial

practice a case study Empirical Software Engineering 22(6)2937ndash2971 Dec 2017[14] E Alegroth Z Gao R Oliveira and A Memon Conceptualization and evaluation

of component-based testing unified with visual gui testing An empirical studyIn 2015 IEEE 8th International Conference on Software Testing Verification andValidation (ICST) pages 1ndash10 April 2015

[15] G Bavota M Linares-Vaacutesquez C Bernal-Caacuterdenas M Di Penta R Oliveto andD Poshyvanyk The impact of api change- and fault-proneness on the user ratingsof android apps Software Engineering IEEE Transactions on 41(4)384ndash407 April2015

[16] J Brooke SUS A quick and dirty usability scale In P W Jordan B WeerdmeesterA Thomas and I L Mclelland editors Usability evaluation in industry Taylorand Francis London 1996

[17] K Charmaz Constructing Grounded Theory SAGE Publications Inc 2006[18] S R Choudhary M R Prasad and A Orso Crosscheck Combining crawling and

differencing to better detect cross-browser incompatibilities in web applicationsIn Proceedings of the 2012 IEEE Fifth International Conference on Software TestingVerification and Validation ICST rsquo12 pages 171ndash180 Washington DC USA 2012IEEE Computer Society

[19] A Ciurumelea A SchaufelbAtildeČAcircijhl S Panichella and H C Gall Analyzingreviews and code of mobile apps for better release planning In 2017 IEEE 24th In-ternational Conference on Software Analysis Evolution and Reengineering (SANER)pages 91ndash102 Feb 2017

[20] A Di Sorbo S Panichella C V Alexandru J Shimagaki C A Visaggio G Can-fora and H C Gall What would users change in my app summarizing appreviews for recommending software changes In Proceedings of the 2016 24thACM SIGSOFT International Symposium on Foundations of Software EngineeringFSE 2016 pages 499ndash510 New York NY USA 2016 ACM

[21] K Holl and F Elberzhager A mobile-specific failure classification and its usageto focus quality assurance In 2014 40th EUROMICRO Conference on SoftwareEngineering and Advanced Applications pages 385ndash388 Aug 2014

[22] G Hu X Yuan Y Tang and J Yang Efficiently effectively detecting mobileapp bugs with appdoctor In Proceedings of the Ninth European Conference onComputer Systems EuroSys rsquo14 pages 181ndash1815 New York NY USA 2014ACM

[23] A Issa J Sillito and V Garousi Visual testing of graphical user interfaces Anexploratory study towards systematic definitions and approaches In 2012 14thIEEE International Symposium on Web Systems Evolution (WSE) pages 11ndash15 Sept2012

[24] N Jones Seven best practices for optimizing mobile testing efforts TechnicalReport G00248240 Gartner

[25] K Kuusinen and T Mikkonen Designing user experience for mobile apps Long-term product owner perspective In 2013 20th Asia-Pacific Software EngineeringConference volume 1 of APSECrsquo13 pages 535ndash540 Dec 2013

[26] V Lelli A Blouin and B Baudry Classifying and qualifying gui defects In 2015IEEE 8th International Conference on Software Testing Verification and Validation(ICST) pages 1ndash10 April 2015

[27] M Linares-Vaacutesquez G Bavota C Bernal-Caacuterdenas M Di Penta R Oliveto andD Poshyvanyk Api change and fault proneness A threat to the success ofandroid apps In Proceedings of the 2013 9th Joint Meeting on Foundations ofSoftware Engineering ESECFSErsquo13 pages 477ndash487 New York NY USA 2013ACM

[28] M Linares-Vaacutesquez G Bavota M D Penta R Oliveto and D Poshyvanyk Howdo API changes trigger Stack Overflow discussions a study on the android SDK

In Proceedings of the 22nd International Conference on Program ComprehensionICPCrsquo14 pages 83ndash94 2014

[29] M Linares-Vaacutesquez G Bavota M Tufano K Moran M Di Penta C VendomeC Bernal-Caacuterdenas and D Poshyvanyk Enabling mutation testing for androidapps In Proceedings of the 2017 11th Joint Meeting on Foundations of SoftwareEngineering ESECFSE 2017 pages 233ndash244 New York NY USA 2017 ACM

[30] S Mahajan A Alameer P McMinn and W G Halfond Automated repair oflayout cross browser issues using search-based techniques In InternationalConference on Software Testing and Analysis ISSTArsquo17 2017

[31] S Mahajan and W G J Halfond Detection and localization of html presentationfailures using computer vision-based techniques In Proceedings of the 8th IEEEInternational Conference on Software Testing Verification and Validation ICSTrsquo15April 2015

[32] S Mahajan B Li P Behnamghader and W G Halfond Using visual symptomsfor debugging presentation failures in web applications In Proceeding of the9th IEEE International Conference on Software Testing Verification and Validation(ICST) ICSTrsquo16 April 2016

[33] T McDonnell B Ray andM Kim An empirical study of api stability and adoptionin the android ecosystem In Proceedings of the 2013 International Conference onSoftware Maintenance ICSMrsquo13 pages 70ndash79 2013

[34] J Meskens K Luyten and K Coninx Plug-and-design Embracing mobiledevices as part of the design environment In Proceedings of the 1st ACM SIGCHISymposium on Engineering Interactive Computing Systems EICS rsquo09 pages 149ndash154 New York NY USA 2009 ACM

[35] K Moran B Li C Bernal-Caacuterdenas D Jelf and D Poshyvanyk Gvt onlineappendix httpwwwandroid-dev-toolscomgvt

[36] P Morville User experience design httpsemanticstudioscomuser_experience_design

[37] B Myers Challenges of hci design and implementation interactions 1(1)73ndash83Jan 1994

[38] B N Nguyen B Robbins I Banerjee and A Memon Guitar An innovativetool for automated testing of gui-driven software Automated Software Engg21(1)65ndash105 Mar 2014

[39] T A Nguyen and C Csallner Reverse engineering mobile application userinterfaces with REMAUI In Proceedings of the 2015 30th IEEEACM Interna-tional Conference on Automated Software Engineering ASErsquo15 pages 248ndash259Washington DC USA 2015 IEEE Computer Society

[40] F PalombaM Linares-Vaacutesquez G Bavota R Oliveto M D Penta D Poshyvanykand A D Lucia User reviews matter tracking crowdsourced reviews to supportevolution of successful apps In 2015 IEEE International Conference on SoftwareMaintenance and Evolution (ICSME) pages 291ndash300 Sept 2015

[41] F Palomba P Salza A Ciurumelea S Panichella H Gall F Ferrucci andA De Lucia Recommending and localizing change requests for mobile appsbased on user reviews In Proceedings of the 39th International Conference onSoftware Engineering ICSE rsquo17 pages 106ndash117 Piscataway NJ USA 2017 IEEEPress

[42] S Roy Choudhary M R Prasad and A Orso X-pert Accurate identification ofcross-browser issues in web applications In Proceedings of the 2013 InternationalConference on Software Engineering ICSE rsquo13 pages 702ndash711 Piscataway NJUSA 2013 IEEE Press

[43] S Roy Choudhary H Versee and A Orso Webdiff Automated identification ofcross-browser issues in web applications In Proceedings of the 2010 IEEE Interna-tional Conference on Software Maintenance ICSM rsquo10 pages 1ndash10 WashingtonDC USA 2010 IEEE Computer Society

[44] T Silva da Silva A Martin F Maurer and M Silveira User-centered design andagile methods A systematic review In Proceedings of the 2011 Agile ConferenceAGILE rsquo11 pages 77ndash86 Washington DC USA 2011 IEEE Computer Society

[45] K-J Stol P Ralph and B Fitzgerald Grounded theory in software engineeringresearch A critical review and guidelines In Proceedings of the 38th InternationalConference on Software Engineering ICSE rsquo16 pages 120ndash131 New York NY USA2016 ACM

[46] A B Tucker Computer Science Handbook Second Edition Chapman amp HallCRC2004

[47] Q Xie M Grechanik C Fu and C Cumby Guide A gui differentiator In 2009IEEE International Conference on Software Maintenance ICSMrsquo09

[48] H Yee S Pattanaik and D P Greenberg Spatiotemporal sensitivity and visualattention for efficient rendering of dynamic environments ACM Trans Graph20(1)39ndash65 Jan 2001

[49] C Zeidler C Lutteroth W Stuerzlinger and G Weber Evaluating Direct Manip-ulation Operations for Constraint-Based Layout pages 513ndash529 Springer BerlinHeidelberg Berlin Heidelberg 2013