Lessons learnt from using DSLs for Automated Software Testing ?· Lessons learnt from using DSLs for…

Download Lessons learnt from using DSLs for Automated Software Testing ?· Lessons learnt from using DSLs for…

Post on 10-Jun-2018




0 download

Embed Size (px)


  • Lessons learnt from using DSLsfor Automated Software Testing

    Mark Micallef, Christian Colombo{mark.micallef | christian.colombo}@um.edu.mt

    PEST Research LabFaculty of Information & Communication Technology

    University of Malta

    AbstractDomain Specific Languages (DSLs) provide a meansof unambiguously expressing concepts in a particular domain.Although they may not refer to it as such, companies buildand maintain DSLs for software testing on a day-to-day basis,especially when they define test suites using the Gherkin language.However, although the practice of specifying and automating testcases using the Gherkin language and related technologies suchas Cucumber has become mainstream, the curation of such lan-guages presents a number of challenges. In this paper we discusslessons learnt from five case studies on industry systems, twoinvolving the use of Gherkin-type syntax and another three casestudies using more rigidly defined language grammars. Initialobservations indicate that the likelihood of success of such effortsis increased if one manages to use an approach which separatesthe concerns of domain experts who curate the language, userswho write scripts with the language, and engineers who wirethe language into test automation technologies thus producingexecutable test code. We also provide some insights into desirablequalities of testing DSLs in different contexts.


    In her widely cited paper about the future of softwaretesting, Bertolino [1] claims that domain specific languages(DSLs) have emerged as an efficient solution towards allowingexperts within a domain to express specifications in thatdomain. She goes on to claim that success of domain-specificapproaches should be built upon and extended to the testingstage of software engineering. An intuitive place to start wouldbe to explore DSLs in the context of automated software test-ing such that languages constructed by domain experts can beleveraged to specify not only requirements but also test caseswhich validate those requirements. This ties in nicely with theindustrys ever growing appetite for automated testing. To alarge extent, the use of DSLs for specifying automated testsis already widespread in the software development industrythrough the use of Gherkin [2] a language designed toexpress test scenarios and is often referred to as the Given-When-Then notation:

    Given I am a userWhen I log in using valid credentialsThen I should see the welcome page

    2015 IEEE Eighth International Conference on Software Testing, Verifica-tion and Validation Workshops (ICSTW)10th Testing: Academic and Industrial Conference - Practice and ResearchTechniques (TAIC PART)978-1-4799-1885-0/15/$31.00 c2015 IEEE

    The language provides constructs for defining a numberof scenarios which in turn consist of a number of steps, eachbeing either a precondition to the scenario (Given) followed bya stimulus to the system (When) and finally the postcondition(Then). In the software development industry, hundreds ofsuch scenarios are developed in a typical project and are usedas a vehicle for communicating specifications amongst teammembers and eventually end up being turned into automatedtesting code via technologies such as Cucumber [2]. Thelanguage is very flexible in that it only requires steps to startwith one of the given-when-then keywords whilst the rest ofthe step is specified as free text by the user of the language.Developers then write code which translates each step intointeraction with the system, thus creating an automated testsuite. In essence, the collection of phrases after a given-when-then keyword form a domain specific language as defined bywhoever is maintaining scenarios written in Gherkin.

    In this paper we present observations from five case studiesinvolving the use of domain specific languages. The first twocase studies consisted of working with two industry partners1and introducing Gherkin-driven test automation frameworks.Whilst the industry partners were interested in setting upautomated testing frameworks, we were mainly interested inobserving the challenges encountered when using Gherkin todevelop a domain specific language. We subsequently went onto investigate the possibility of using more rigorously definedlanguages for specifying automated tests through three casestudies: (1) Android applications, (2) eCommerce websites and(3) graphics-based games.


    This section provides a brief background of domain specificlanguages and automated testing the two main subject areasof this paper.

    A. Domain Specific Languages

    Fowler and Parsons define Domain Specific Languages(DSLs) as computer programming languages of limited ex-pressiveness focused on a particular domain [3]. They arechampioned as a mechanism to improve expressiveness withina domain and have been applied to a wide variety of applica-tions in software engineering.

    1One partner was an online gaming company whilst the other was aninternational publishing house.

  • The literature contains numerous guidelines to guide DSLdesign [3][4][5][6][7] but if one consolidates the sources, anumber of common characteristics of a good DSL emerge.Firstly, all sources strongly argue that a DSL should ex-hibit simplicity in that it is easily understood by humansyet parseable by machines. Related to this, a DSL should,where possible, exhibit similarity to another language withwhich users are already familiar (e.g. English). This contributessignificantly to the initial acceptance and long-term adoptionof the language. Thirdly, a DSL should be highly domainspecific and parsimonious so that any notions not related to thedomain in question are omitted; yet at the same time languagedesigners should strive to make a language as complete aspossible with respect to that domain. Finally, a domain specificlanguage should ideally be extensible thus making it easy toadd features and operations to the language; and reusable inthat the same grammar can be used as the basis of a newlanguage when required.

    B. Automated Testing

    Test automation technologies can be split into two broadcategories: record-playback and scripted automation [8]. Theformer involves using recording software which observes asystem being used and then simply replaying a series of actionsback when required; whilst the latter involves someone writingcode which when executed interacts with the system under testas part of an automated test suite. In this work, we are mainlyinterested in scripted testing.

    There are a number of tools/APIs available which enableprogrammatic interaction with a system under test to facilitatetest scripting. The Selenium/Webdriver [9] project is an opensource endeavour which has become the de facto industrystandard for web test automation. The project provides APIbindings for a variety of programming languages and enablesdevelopers to launch and control web browsers for the purposesof automated testing. The technology interacts with a websiteunder test by querying and interacting with the browser andthe document-object model (DOM) of the website. Two relatedtechnologies which build on the Selenium/Webdriver projectare Selendroid [10] and Appium [11]; two technologies whichallow automated interaction and querying of mobile devicesusing a DOM-based querying mechanism. Finally, for systemswhich do not expose their internal representation of the userinterface, Sikuli [12] is an automated testing API which utilisesimage recognition techniques. That is to say, instead of (forexample) searching through a websites DOM looking for abutton with the text OK, Sikuli takes a screenshot of theapp and looks for an image of such a button. The technologyis slower and more brittle than the alternatives but in certainsituations, it is the only option available. We discovered thiswhen attempting to automate the testing of graphics-basedgames.

    Whilst automating the execution of tests is becoming moreand more popular [13], with some considering it a necessity[14], test automation efforts tend to be expensive and suscep-tible to losing steam as a project grows and evolves in waysthat makes maintaining old tests difficult. In this paper weargue that hiding the low level intricacies of test automationtechnologies within a domain specific language goes a longway towards improving the usability of the technology, mainly

    Fig. 1. The layered approach proposed as a guideline for developing a DSLfor software testing

    through separating the role of the domain expert who curatesthe DSL and the engineer who wires phrases of the languageinto test automation tools. This is discussed further in thefollowing section.


    Our initial steps on this journey consisted of two casestudies with industry partners, one in the iGaming industry andthe other was an international publishing house. We embeddeda researcher with each of the industry partners for a period ofthree months with the researcher forming part of a productdevelopment team in a test engineer role. The arrangementwas mutually beneficial in that the partners would benefit fromthe services of an embedded researcher with test automationexperience who would champion and drive the creation of anautomated testing framework, whilst the research group wasable to observe challenges which arose throughout the process.

    Following initial discussions, two main problems wereidentified. Firstly, whilst test automation was the primaryoutcome sought by our partners, they acknowledged that theirtesters did not necessarily have the required skills to implementautomated tests. Whilst an effort would be made to recruitexperienced technical testers, the balance of technical to non-technical testers was unlikely to ever be close to even. Thesecond problem, stated bluntly, was that delivery deadlineswere a reality and sometimes tes