development of eyk - skemman · facultyofindustrialeng,mechanicaleng.andcomputerscience...

Faculty of Industrial Eng, Mechanical Eng. and Computer ScienceUniversity of Iceland

2015

Faculty of Industrial Eng, Mechanical Eng. and Computer ScienceUniversity of Iceland

2015

Development of EYK

Helgi Gylfason

DEVELOPMENT OF EYK

Helgi Gylfason

60 ECTS thesis submitted in partial fulfillment of aMagister Scientiarum degree in Software Engineering

AdvisorHelmut Neukirchen

Steinn Guðmundsson

Faculty RepresentativeHeiðar Einarsson

Faculty of Industrial Eng, Mechanical Eng. and Computer ScienceSchool of Engineering and Natural Sciences

University of IcelandReykjavik, 10 2015

Development of EYKDevelopment of EYK, a data management system for geographically distributed sources.60 ECTS thesis submitted in partial fulfillment of a M.Sc. degree in Software Engineering

Copyright c© 2015 Helgi GylfasonAll rights reserved

Faculty of Industrial Eng, Mechanical Eng. and Computer ScienceSchool of Engineering and Natural SciencesUniversity of IcelandDunhagi 5107, Reykjavik, ReykjavikIceland

Telephone: 525 4000

Bibliographic information:Helgi Gylfason, 2015, Development of EYK, M.Sc. thesis, Faculty of Industrial Eng, MechanicalEng. and Computer Science, University of Iceland.

Printing: Háskólaprent, Fálkagata 2, 107 ReykjavíkReykjavik, Iceland, 10 2015

ForRut, Ísold and Nói

Abstract

EYK is a data processing system that has been in development for two and a half year.The system started as a replacement for an in-house system at Reykjavik Energy (RE)but grew to a larger system once an idea arose to service EYK to a larger market.

EYK is developed as a web application to be accessible from most type of devices. Itis designed in a modular way so that new features can be easily added. The system isalso capable of connecting to any data source a customer has access to and thereforethe customer can continue using existing data sources with EYK. To establish if othercompanies could use EYK the Icelandic energy market was studied and the results werethat other companies have much use for EYK. The system was then supposed to be soldthrough one-time payments and subscription cost. This has changed and a simpler salesplan is used. The customer now hands in a feature request and receives a cost estimate inreturn and once an agreement is reached the feature is implemented.

The team behind EYK consists of only two members and they need to handle varioustasks ranging from business meetings to creating test scripts. In order to make it possiblefor them to develop EYK at a sufficient pace it is critical for them to be well organizedand minimize work outside software development. The original work-flows for the teamwas built on Scrum but as the project went along the work-flows were simplified and someparts of Scrum such as sprints were removed as they did not work well for such a smallteam.

Tests were neglected in the beginning of the project but as the project went along thanhigher credence was given to tests. The goal is then to eventually use test driven devel-opment where automated test are run each time when code is added to EYK. The teambehind does not maintain a physical server as that is provided by the customer. Theyhowever need to maintain the Operating System (OS) of the server and they do that usinga configuration management tool called Ansible. It can automate recurring tasks such asdeployment, setting up developer machine or building test servers so that the developerscan spend more time improving EYK.

v

Contents

List of Figures ix

List of Tables xi

List of Listings xi

Abbreviations xv

Acknowledgments xvii

1. Introduction xix1.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

2. Background and requirements 32.1. Gagnor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.2. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3. The EYK design 93.1. Base . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.1.1. Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.2. Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3.2.1. Current modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.2.2. Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.3. Database interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.4. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4. Business model 214.1. Energy market . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

4.1.1. Geothermal plants . . . . . . . . . . . . . . . . . . . . . . . . . . . 224.1.2. Hydro power plants . . . . . . . . . . . . . . . . . . . . . . . . . . . 244.1.3. District heating and water facilities . . . . . . . . . . . . . . . . . . 244.1.4. Power transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254.1.5. Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274.1.6. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4.2. Sales model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274.2.1. Original sales model . . . . . . . . . . . . . . . . . . . . . . . . . . 274.2.2. Current sales model . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4.3. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

5. Technologies 295.1. Web Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

vii

Contents

5.2. Map software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305.3. Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315.4. Web Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325.5. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

6. Development processes 336.1. Project Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

6.1.1. Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336.1.2. The work flows at Hugfimi . . . . . . . . . . . . . . . . . . . . . . . 34

6.2. Source control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386.2.1. Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386.2.2. Branching model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396.2.3. Versioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

6.3. Configuration management . . . . . . . . . . . . . . . . . . . . . . . . . . . 406.3.1. Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 416.3.2. Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426.3.3. Example: Deploying to demo server . . . . . . . . . . . . . . . . . . 43

6.4. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

7. Testing 497.1. Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 497.2. Unit and integration testing . . . . . . . . . . . . . . . . . . . . . . . . . . 51

7.2.1. Test case: datebar . . . . . . . . . . . . . . . . . . . . . . . . . . . . 547.3. System testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

7.3.1. Test case: modules for an air quality monitor . . . . . . . . . . . . . 587.4. Future improvements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 627.5. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

8. Conclusions 678.1. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 678.2. Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

References 69

A. Django 75A.1. Apps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

A.1.1. Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76A.1.2. Views . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77A.1.3. Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

B. Datebar refactored: Full code 81

C. Ansible 85

D. Deployment to demo server: Full example 89D.1. Role: common . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91D.2. Role: apache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92D.3. Role: postgres . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95D.4. Role: django . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97D.5. Role: demo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

viii

List of Figures

2.1. Choosing programs in Gagnor. . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.2. GUI for borehole measurements data program in Gagnor. . . . . . . . . . . 4

2.3. A plot of borehole measurement data in Gagnor. . . . . . . . . . . . . . . . 6

3.1. A diagram of EYK’s design. . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.2. The map interface of EYK . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.3. The top bar on the asset page with links to enabled modules. . . . . . . . . 11

3.4. A zone in EYK spanning an area containing multiple boreholes. . . . . . . 12

3.5. An example of a properties module . . . . . . . . . . . . . . . . . . . . . . 14

3.6. An example of measurement series module . . . . . . . . . . . . . . . . . . 15

3.7. An example of a time series module . . . . . . . . . . . . . . . . . . . . . . 16

3.8. An example of a data monitoring module (prototype) . . . . . . . . . . . . 17

4.1. IMO’s water sensor network [Ís12] . . . . . . . . . . . . . . . . . . . . . . . 24

4.2. An example of RARIK’s street cabinet and its content . . . . . . . . . . . 26

5.1. A map interface for “Já.is” with many features. . . . . . . . . . . . . . . . . 31

6.1. Product backlogs in Trello and Jira . . . . . . . . . . . . . . . . . . . . . . 36

6.2. A sprint backlog in Trello. . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

6.3. The sprint work flow for the first iteration . . . . . . . . . . . . . . . . . . 38

6.4. A JIRA board. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

ix

LIST OF FIGURES

6.5. Git flow branching model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

7.1. The GUI of the datebar . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

7.2. The property module for an air quality monitor . . . . . . . . . . . . . . . 58

7.3. Time series module for an air quality monitor . . . . . . . . . . . . . . . . 60

7.4. A flowchart for Behaviour Driven Development (BDD). . . . . . . . . . . . 63

C.1. How Hugfimi connects to customer’s instances of EYK . . . . . . . . . . . 85

D.1. The start page for the EYK demo. . . . . . . . . . . . . . . . . . . . . . . 104

x

List of Tables

2.1. The four parts of RE’s ID code for data sources. . . . . . . . . . . . . . . . 5

4.1. Companies inside the energy sector of Iceland and their area of operations. 21

4.2. Estimations for modules and extensions separated by sections . . . . . . . 22

4.3. Number of modules for geothermal data-sets . . . . . . . . . . . . . . . . . 23

4.4. Estimate for number of modules for geothermal . . . . . . . . . . . . . . . 23

xi

Listings

6.1. Vagrantfile: Configurations for the virtualbox . . . . . . . . . . . . . . . . . 416.2. Vagrantfile: Configurations for networking . . . . . . . . . . . . . . . . . . 416.3. Ansible project layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436.4. Ansible host file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436.5. The main playbook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 446.6. The roles used for configuring the demo server . . . . . . . . . . . . . . . . 446.7. The group variables for the demo server . . . . . . . . . . . . . . . . . . . 456.8. List of subroles for the Django role . . . . . . . . . . . . . . . . . . . . . . 466.9. Ansible tasks for the git part in the Django role . . . . . . . . . . . . . . . 466.10. Ansible handler for the git part in the Django role . . . . . . . . . . . . . . 466.11. Ansible template for Django settings in production environment . . . . . . 47

7.1. The Django models for assets . . . . . . . . . . . . . . . . . . . . . . . . . 517.2. Mocks for Django models . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527.3. Using mocks in testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527.4. Testing a Django view . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537.5. Datebar closure created . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 547.6. Closures nested in the datebar . . . . . . . . . . . . . . . . . . . . . . . . . 557.7. Private closures in datebar made public. . . . . . . . . . . . . . . . . . . . 557.8. Simple unit test for datebar. . . . . . . . . . . . . . . . . . . . . . . . . . . 557.9. Example of how SinonJS is used for manually setting date . . . . . . . . . 567.10. Datebar refactored for testability . . . . . . . . . . . . . . . . . . . . . . . 577.11. Closures inside Datebar refactored . . . . . . . . . . . . . . . . . . . . . . . 577.12. System testing for air quality monitor: checking page load . . . . . . . . . 597.13. System testing for air quality monitor: checking available modules . . . . . 597.14. System testing for air quality monitor: checking asset ID . . . . . . . . . . 597.15. System testing for air quality monitor: check if time series module loads . . 617.16. System testing for air quality monitor: check graph controls . . . . . . . . . 617.17. System testing for air quality monitor: assessing graph control behaviour . 617.18. System testing for air quality monitor: check plot output . . . . . . . . . . 627.19. System testing for air quality monitor: check table . . . . . . . . . . . . . . 627.20. Example of a use case in Behave: feature description . . . . . . . . . . . . 647.21. Example of a use case in Behave: test code . . . . . . . . . . . . . . . . . . 64

A.1. Django project layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75A.2. Example code for URL dispatcher . . . . . . . . . . . . . . . . . . . . . . . 76A.3. Example code for view matching URL request . . . . . . . . . . . . . . . . 76A.4. Django app layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77A.5. Example of a Django model . . . . . . . . . . . . . . . . . . . . . . . . . . 77A.6. Example of Django view logic . . . . . . . . . . . . . . . . . . . . . . . . . 78

xiii

Listings

A.7. Example of a Django template . . . . . . . . . . . . . . . . . . . . . . . . . 79

B.1. Datebar before refactoring . . . . . . . . . . . . . . . . . . . . . . . . . . . 81B.2. Datebar after refactoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

C.1. Ansible project layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86C.2. The inventory file (hosts) for Ansible project . . . . . . . . . . . . . . . . . 86C.3. Variables inside an inventory file . . . . . . . . . . . . . . . . . . . . . . . . 86C.5. The main playbook for EYK demo . . . . . . . . . . . . . . . . . . . . . . 87

D.1. Ansible host file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89D.2. The group variables for the demo server . . . . . . . . . . . . . . . . . . . 89D.3. The roles used for configuring the demo server . . . . . . . . . . . . . . . . 90D.4. Ansible tasks for the common role . . . . . . . . . . . . . . . . . . . . . . . 91D.5. Ansible tasks for the apache role: install Apache . . . . . . . . . . . . . . . 92D.6. Ansible tasks for the apache role: Configuring user apache . . . . . . . . . 92D.7. Ansible tasks for the apache role: Settings up sites . . . . . . . . . . . . . . 93D.8. Ansible variables for the apache role . . . . . . . . . . . . . . . . . . . . . . 93D.9. Ansible handlers for the Apache role . . . . . . . . . . . . . . . . . . . . . 94D.10.Ansible tasks for the postgres role . . . . . . . . . . . . . . . . . . . . . . . 95D.11.Ansible tasks for the postgres role: Setting up user postgres . . . . . . . . . 96D.12.Ansible tasks for the postgres role: Initialize database cluster . . . . . . . . 96D.13.Ansible tasks for the postgres role . . . . . . . . . . . . . . . . . . . . . . . 96D.14.List of subroles for the Django role . . . . . . . . . . . . . . . . . . . . . . 97D.15.Ansible tasks for the git part in the django role . . . . . . . . . . . . . . . 97D.16.Ansible handler for the git part in the Django role . . . . . . . . . . . . . . 98D.17.Ansible template for Django settings in production environment . . . . . . 98D.18.Ansible tasks for the YUM part in the Django role . . . . . . . . . . . . . 99D.19.Ansible tasks for the pip part in the Django role . . . . . . . . . . . . . . . 100D.20.Ansible tasks for the database part in the Django role . . . . . . . . . . . . 101D.21.Ansible tasks for Django part in the django role . . . . . . . . . . . . . . . 102D.22.Ansible tasks for the common part in the django role . . . . . . . . . . . . 102D.23.Ansible tasks for the demo role . . . . . . . . . . . . . . . . . . . . . . . . 103

xiv

Abbreviations

BDD Behaviour Driven Development

CLI Command-Line Interface

CSV Comma-Separated Values

EPS Encapsulated PostScript

GNS Global Namespace

GUI Graphical-User-Interface

ICA the Iceland Construction Authority

Landsvirkjun The national energy company of Iceland

MTV Model-Template-View

MVC Model-View-Controller

NoSQL Not-Only-SQL

RE Reykjavik Energy

NEA the National Energy Authority

ORM Object-Relational-Mapping

OS Operating System

PDF Portable Document Format

PNG Portable Network Graphics

RARIK the Iceland State Electricity

xv

Listings

RDBMS Relational database management system

SSH Secure-Shell

TMS Tile-Map-Service

IMO the Icelandic Met Office

VM Virtual Machine

VPN Virtual Private Network

xvi

Acknowledgments

I want to thank my friend and colleague Bjarki Ásbjarnarson for taking this journey withme in making EYK. I also want to thank my advisors: Helmut Neukirchen and SteinnGuðmundsson for their contribution to EYK and this thesis. A special thank to HeiðarEinarsson for giving his time and effort to review this thesis. Lastly but not least I wantto thank my family for giving me the support needed to finish this thesis and to get EYKto where it is now.

xvii

1. Introduction

Icelandic energy companies collect a lot of data and store it in various types of data sources.When the data is later analyzed it is very important that it is easily accessible howeverthat is not always the case. It can take a person a long time to collect the necessary dataespecially in the case when it is distributed across multiple data sources. EYK is a webapplication which addresses this problem in two ways. Firstly it can connect to multipledata sources, fetch their data and store it in structured form in a database. Secondly itvisualizes the data from its database to enable the user to easily interpret the data.

EYK has been in continuous development by the author of this thesis together withBjarki Ásbjarnarson. They work together under the group name Hugfimi. Bjarki workedpreviously at RE in developing and maintaining data processing systems. It became evidentto him that these systems were not taking advantage of new technologies and thereforehe saw an opportunity for creating a better system. He presented the idea to me andwe developed it further and came to the conclusion that we could have full time jobs inservicing this new system.

For a software solution to succeed it is not enough to create quality software as it mustserve a meaningful purpose, otherwise nobody becomes interested in using it. Therefore inthe beginning of a project, a good amount of work should be put into finding the purposeand demand for the solution and once that is complete the software development shouldstart. Software development is normally done in teams which makes it important to havepredefined work-flows. They ensure that tasks are handled correctly and facilitate for betterknowledge management. Already there exist guidelines and ideologies for how a team’swork-flow should be but given the unconventionally small team working on EYK thesemethods need to be reevaluated. In addition to team work, the source code of EYK needsto be kept organized so that changes from team members will not collide and break thesoftware. But even in those cases where the software breaks, good source control enablesit to be reverted to an earlier functional state.

One does not though want to break the software often and a way to prevent that is to testthe software regularly and thoroughly. Once the software has been tested it has yet to bedelivered to the customer. In some cases the software only needs to be delivered to onedevice but in EYK’s case it has to be delivered to a separate device for each customer. Inthat case the time spent on delivering updates increases with each new customer whichfurther supports the requirement for an efficient work-flow.

It is evident that there are many issues to be addressed when developing a software. Thisthesis describes how a team size of two is able to handle these issues for EYK.

xix

1.1. Overview

1.1. Overview

Following this introduction, chapter 2 describes the background of the project and therequirements derived from the current system in use at RE. Chapter 3 provides an overviewof the design of EYK. In chapter 4 the target market, i.e. Icelandic energy market, and thesales model is discussed. Chapter 5 describes technologies used for developing developingEYK and chapter 6 discusses the software development processes. Software testing is thetopic of chapter 7 and in chapter 8 the thesis is summarized and an outlook is provided.

1

2. Background and requirements

The idea of EYK came about from work on developing Gagnor, a data processing systemfor RE. Gagnor handles various types of data that are collected from spatially distributedsources e.g. boreholes and power plants. In this chapter Gagnor is described and itsstrengths and weaknesses are evaluated in order to set requirements for EYK. When theserequirements are met, EYK can be considered a successful replacement for Gagnor.

2.1. Gagnor

Gagnor is a collection of data processing programs that are made accessible through aunified Graphical-User-Interface (GUI) like shown in fig. 2.1. Gagnor’s programs connectto various types of data-sets. Each program is customized to a specific data collectionand provides customized calculations and presentation of its data. The flexibility forcustomization is a consequence of that the programs are independent from each otherhowever a lot of duplicate code has been created to keep them independent, increasingmaintenance cost.

Figure 2.1: Choosing programs in Gagnor.

Gagnor runs on a centralized Linux server that users have to log into remotely. Users of aMicrosoft OS, which most at RE are, need to use a third party tool Xming [Har14], a XWindow System client to connect to the server. Once users have been logged in they must

3


navigate around a Linux environment to access the programs. This process is tedious formost users as they are only familiar with Microsoft Operating Systems. Additionally forsecurity reasons the server is only available on RE’s intranet. As data sources are locatedoutside the intranet their data is transported in a physical format to where the connectionis possible and then inserted into Gagnor. Even though it is not done it would be possibleto connect to Gagnor outside the intranet but that requires a Virtual Private Network(VPN) connection. Improved accessibility is an important requirement for EYK as usersshould be able to access EYK as easily as they do other web applications.

To demonstrate the functionality of Gagnor it is best to use one of its programs as an ex-ample. The program for borehole measurements was chosen, it handles data measurementsfor temperature and pressure in boreholes. The data can either be plotted or exported ina simple text format but before that can be done the user must first identify the correctdata source using the interface shown in fig. 2.2.

Figure 2.2: GUI for borehole measurements data program in Gagnor.

In the figure there are four columns and each of them represent a part of a 13 letter IDcode that is used for all data sources in Gagnor. In table 2.1 the parts of the ID codeare summarized using the borehole as an example. The first part of the code is the zonein which the data source is located. In the case of the borehole chosen in fig. 2.2 thezone (“HHE”) stands for Hellisheidi, an area located just outside Reykjavik in which REoperates a geothermal power station. The next part, category, indicates the type of thedata source which in the case of the borehole is a production hole (“VE”). The thirdpart is called place and is the name of the data source and HE003 is the name for the

4

2.1. Gagnor

chosen borehole. Final part of the code is the subplace but it identifies if the data sourceis positioned at the top or bottom of the borehole. In the example the code is HOT whichis at the top of the borehole.

Table 2.1: The four parts of RE’s ID code for data sources.

Zone Category Place Sub-Place

HHE VH HE003 HOTHellisheidi production hole hole HE003 top of hole

It has proven to be difficult for inexperienced users to identify data sources using thiscode and mistakes have been made. In the worst cases the data is registered under awrong data source. This type of mistake is hard to correct because the data is stored insimple text files and directories are used to categorize the data between data sources. Amisplaced data is therefore put in a wrong directory with no information about its correctdata source. Also this kind of a mistake is only discovered if the misplaced data is differentfrom the rest of the data it was placed with. That is not always the case e.g. if data forsimilar boreholes get mixed there is no clear evidence that the data is wrong. For theabove mentioned problems it is crucial that EYK offers an intuitive way to identify datasources.

Once the correct data source has been found the data can be represented graphically. Incase for the borehole measurements program it involves visualizing the data set with asimple linear graph with temperature plotted against depth (see fig. 2.3).

The plots in Gagnor’s programs are available in two formats, PNG and EPS, both whichare suitable for inclusion in reports. Neither format however enables changes to the plotonce it has been rendered and that limits the user’s ability to explore and analyze the data.Enabling interaction with the plot opens up for many possibilities. One use case being toquickly zoom on data range with abnormal activity and see remarks for the data points onmouse over that explain the strange behavior. EYK is in good position to offer interactiveand feature-rich data presentations as it belongs to a popular development platform (webplatform) that has an enormous range of tools available.

5


Figure 2.3: A plot of borehole measurement data in Gagnor.

2.2. Summary

Gagnor is the current data processing system at RE and EYK is aimed as its successor.The functionality of Gagnor is evaluated in order to realize the demands that will be set onEYK. The system offers customized processing of data and has various types of databaseconnections but with this flexibility comes a lot of duplicate code that has a negative effecton the maintenance cost. Access to Gagnor is limited as it runs on a Linux server that isrestricted to RE’s intranet and requires a tedious process to log into. Gagnor also has anunintuitive process to identify data sources and its presentation presentation of data islimited to a non-interactive format.

A list of requirements for EYK can be made based on the analysis of issues related toGagnor:

• Gagnor is limited to RE’s intranet and has a tedious login process.

1. Access to EYK needs to be simple and not require installation of new thirdparty tools.

• The process of identifying data sources is error prone and inefficient.

2. Identification of data sources in EYK should be intuitive.

6

2.2. Summary

• Gagnor’s data presentation is limited to a non-interactive format.

3. Presentation of data in an interactive format should be available in EYK.

• Gagnor is a collection of independent programs that facilitate customization.

4. EYK needs to be able to meet very specific demands for data processing.

• Gagnor connects to a range of different databases.

5. EYK should offer connection to existing customer databases.

Chapter 3 explains how EYK meets these requirements with its design. EYK’s currentand future features are also described and use cases are found for them when the targetmarket is researched in chapter 4.

7

3. The EYK design

EYK needs to be easily accessible and not require installation of new third party tools (seesection 2.2). The former is achieved by developing EYK as a web application because usersare familiar with using websites and the web platform works on multiple types of devicesand OSs. The latter however is not completely met as a web application requires a webbrowser to be installed. It is however reasonable to assume that a user already have oneavailable on their device and therefore a requirement to use one should not have negativeaffects on accessibility.

The system is split into three parts: database interfaces, modules and base. A diagram ofthe arrangement is shown in fig. 3.1. A database interface connects to customer’s databases,retrieves their data and inserts into EYK’s database. A module processes and presents thedata from the data sources (a data source is called an asset in EYK). The base containsthe map interface that enables users to find an asset. It also contains the asset page whichprovides the data of the asset from the EYK database to its modules. The asset page finallyacts as a container for the output of the modules so that they can operate in isolationfrom each other. Each of these three parts is explained in more detail in following sections.

Figure 3.1: A diagram of EYK’s design.

9

3. The EYK design

3.1. Base

The main page of EYK is a map interface shown in fig. 3.2. As all assets have a uniquelocation it is logical to use a map to provide an intuitive way to identify them. Users canpan and zoom to navigate to an asset instead of having to remember cryptic ID codes.For those who want to use the ID codes a search bar is available.

Figure 3.2: The map interface of EYK

An asset has one type and zero to many asset groups and for each type there is a uniquemap icon. In fig. 3.2 there are two visible icons and both are red and stand for the assettype “air quality sensors”. The green and red circles on the map are clusters and representmultiple assets where the number of assets is indicated by the number inside the circle.Also the color and icon of the cluster represent the type of asset it contains. The clustersare used when assets are in such close proximity that it is hard to distinguish betweenthem and as the user zooms into the map the cluster disappears and the assets becomevisible.

When a user clicks on an asset on the map he/she is directed to its asset page. The assetpage contains the modules that have been enabled for the type and groups the selectedasset belongs to.

A large part of the base is to fetch the asset’s modules, set them up on the asset page andlink them to the correct data-set from the EYK database. The enabled modules are thenmade accessible to the user through links in the top bar of the asset page. An example ofa top bar is shown in fig. 3.3 for an asset of type borehole with five enabled modules.

10

3.1. Base

Figure 3.3: The top bar on the asset page with links to enabled modules.

The processing of the asset’s data is handled by the modules themselves and their role isexplored further in section 3.2.

3.1.1. Future work

Several extensions are scheduled for the base part in order to add more value to it. Theseextensions are:

Zones

A zone functions as an asset except it is displayed as a polygon on the map (see fig. 3.4).Its purpose is to represent groups of assets and their collective data e.g. the averagetemperature of the boreholes at Hellisheiði.

11

3. The EYK design

Figure 3.4: A zone in EYK spanning an area containing multiple boreholes.

Users

User accounts are scheduled to be added to EYK and will make the system leaner andmore secure. The user management is broken into three parts:

• Authentication: Users will be registered into EYK and a simple login process isset up so that only registered users have an access.

• Control: User access inside of EYK will be configurable down to a single asset (orzone) so that only data relevant to the user is displayed. User privileges will also beintroduced but it can be used to control what operations a user can perform on aspecific asset, group of assets and/or zone.

• Log/History: User actions will be logged to provide a limited version control ofdata e.g. if a user makes changes on temperature measurements by mistake, he orshe will be able to retrieve the previous data using logs.

Offline mode

Offline mode is the third extension to be scheduled for the base. It enables users to registerand view data on site without Internet connection. The data input would then be registeredto the database when a connection is established again.

12

3.2. Modules

3.2. Modules

When the asset page has been loaded the selected module starts to process the data forthe asset, zone or asset group. The modules are independent from each other and it istherefore possible for an asset to have any combination of modules in its asset page.

Some of the modules in EYK are intended to replace the programs from Gagnor whileothers provide new functionality.

3.2.1. Current modules

Property module

The property module shows the key properties of an asset and a picture of the asset ifavailable. Every asset has this module and it is the shown by default when the user openthe asset page. In fig. 3.5 is an example of the module. In it are the key properties of aborehole e.g. depth, ID code and coordinates.

13

3. The EYK design

Figure 3.5: An example of a properties module

There are two extensions scheduled for the module and they are described in section 3.2.2.

Measurement Series module

The measurement series modules handles data-sets that can be plotted against a commonparameter e.g. time series are data-sets plotted against time. A measurement series is asuper-set of time series as its data can be plotted against any unit. A separate module iscreated for time series to utilize the wide range of functionality specific for them that areavailable in the visualization tools that are used.

14

3.2. Modules

Figure 3.6: An example of measurement series module

In fig. 3.6 each series stands for multiple temperature measurements made at differentdepths in the borehole. The graph’s time range, set of active series and type of interpolationcan be changed interactively. The asset’s raw data is displayed in a table below the graphand is available for download as a CSV file.

Time Series module

The time series module is displayed in fig. 3.7 and is intended for graphical presentation oftime series. It plots the time series for a given time period as an interactive linear graph.The graph’s time range, set of active series and type of interpolation can be changedinteractively. The plot’s raw data are displayed in a table below the graph and is availableto download as a CSV file.

15

3. The EYK design

Figure 3.7: An example of a time series module

Data monitoring module

This module was created specifically for RE at their request. They wanted an applicationthat enabled them to monitor time-related data from multiple assets. The application willbe shown on a large monitor that is located in their office space and it was important thatthe data would be easily readable from the employees workstations. To achieve that thedata is presented in a slide show where each graph has its own slide. The time for whicheach slide is displayed is then configured to the customer’s preference.

The module is designed for time series and uses the time series module to create the graphs.As an example RE’s instance of the module is shown in fig. 3.8, it has four graphs in itsslide show that are cycled through on a 10 second interval.

16

3.2. Modules

Figure 3.8: An example of a data monitoring module (prototype)

3.2.2. Future work

Scheduled extensions for the system are described in the following sections. These exten-sions are either new functionality for existing modules or completely new modules.

Property module

pE1: Commenting will be made possible in the property module once the userauthentication has been introduced into EYK. Users will be able to insert commentsand attach files related to the asset. The users can use the comments to e.g. logwhen sensors are collected and they can attach asset reports to their correspondingasset.

Measurement Series and Time Series module

The extensions scheduled for the measurement and times series modules are the same asthe modules are closely related and are therefore summarized together.

mE1 (measurement series),tE1 (time series): Users will be able to add com-ments to individual measurements. This extension is a response to a request fromRE.

mE2 (measurement series),tE2 (time series) Users will be able to add newdata through the module but currently data is retrieved from customer’s databases.

17

3. The EYK design

Component module

Some assets are large and contain other smaller assets e.g. a power plant can containmultiple machines where each machine has its own set of data. The components modulemakes these smaller assets (e.g. machines) accessible from the larger asset (e.g. powerplant). It does so by displaying them in a list as links to their corresponding asset page.This navigation layer between the map and the asset page enables EYK to provide anintuitive way to access assets that are nested inside other assets.

Asset map module

Asset map module is similar to the component module but differs in how the componentsare displayed. In the asset map module the components are displayed on a map. A usecase for this module would be a power plant where the floor plan of the plant would bethe layer for the map and each component is displayed with an icon on the map. A userwould then simply need to pan to a component and click on it to access its asset page.

Report module

This module is intended to improve a work flow at RE where considerable time is spentcollecting data for annual reports. The report module automates this process and can beused for reports that are prepared regularly by well defined procedures. An example ofthese reports would be the audit reports about changes in the physical state of geother-mal systems which RE are required to submit to the National Energy Authority (NEA)annually [Nat14a]. The data used in the report is from the sensors in certain boreholesthat EYK has access to. The module would use the data to create graphs and tables thatwould be inserted into either a LATEX or Word document that is available for download.If necessary the user can then manually fill the report with his/her text. In cases wherereports are fully standardized a complete report in either PDF or Word format would beavailable for download.

Sample module

A sample is defined as a set of parameters at a single time point. The sample module keepstrack of sample data e.g. chemical samples or trace samples. The first version displays thesamples and enables the user to download the data either as a PDF or a CSV file.

sE1: Users will be able to manually add data through EYK’s interface. This enablese.g. an employee at RE to collect samples and insert their data on site.

sE2: Users will be able to analyze samples in EYK with Watch [Ice14b] which is ahighly specialized analysis tool for geothermal chemical samples. Watch is currentlyused at RE to analyze chemical samples for high and low temperature samples. It is

18

3.3. Database interfaces

run through a Command-Line Interface (CLI) and has a simple text based interface.The aim is to improve this interface when it gets integrated into EYK.

Customization

The customer always has the option to request a customized module in addition to thecurrent and scheduled modules. The developers will estimate the cost and present an offerfor the module.

3.3. Database interfaces

The database interfaces enable EYK to retrieve data from databases a customer hasand store it in the EYK database. It also enables customers to continue using theirexisting databases alongside EYK which makes it easy for them to try out EYK withoutchanging their infrastructure. Initially data was retrieved from the customer’s databaseson user request. This meant that each time a user requested e.g. a plot to be renderedthe corresponding data had to be retrieved from the customer’s database. That routeproved to be both unreliable and slow and therefore a middle step was created. In thismiddle step the data is periodically collected into the EYK database. The user definesthe frequency of the data collecting and choices therefore between minimizing the delayof data availability in EYK or minimizing the load on EYK.

3.4. Summary

EYK has been designed to meet the requirements set in chapter 2. The system meetsthese requirements mainly through its modular design. The system is made accessible ona wide range of devices by building it as a web application. A map interface is used toprovide an intuitive way to identify the data sources (which are called assets in EYK).The processing of asset data is implemented inside modules to enable great customizationand the presentation of data is implemented in an interactive way. Finally the interfaces tothe customer’s databases are kept as independent from the rest of the system as possibleto facilitate for the process of adding new data collections.

In the next chapter a target market is defined and uses cases are found for the organizationson that market. Following that the sales model for EYK will be described.

19

4. Business model

The market for EYK was originally limited to RE however when the idea arose to build abusiness around the solution it became clear that the market had to be expanded. OtherIcelandic energy companies were therefore researched and the results of that research isthe main topic of this chapter. The results describe the operations of the companies andtheir use cases for EYK, both which provide an approximation for their requirements. anestimation for their requirements. Finally the sales model is discussed which explains thepricing of EYK.

4.1. Energy market

Energy companies in Iceland can be divided into three groups: energy production, energytransfer and heating & water supply. Electricity production in the country mainly consistsof hydroelectric (71%) and geothermal (22%) plants [Aut11]. The electricity transfersector is split into transmission and distribution. In table 4.1 a summary is shown of thesecompanies and their operations.

Table 4.1: Companies inside the energy sector of Iceland and their area of operations.

Company Hydro-power Heating Distribution MonitoringGeothermal Water Facilities Transmission

HS-Orka X X X XLandsnet XLandsvirkjun X XNorðurorka X X X XNEA XRE X X X X XRARIK/Orkusalan X X X

The modules and extensions which each sector of the market would use are outlined intable 4.2. "X" means that the module is used without extensions and "-" means thatthe module is not used. "tE1" and "tE2" stand for the extensions of their correspondingmodule listed in previous chapter. The requirements for each sector are then explained inthe following sections.

21

4. Business model

Table 4.2: Estimations for modules and extensions separated by sections

Modules Hydro-power Heating Distribution MonitoringGeothermal Water Facility Transmission

Sample - sE1&2 sE1 sE1 - - sE1Time Series tE1&2 tE1&2 tE1&2 tE1&2 tE1&2 tE1&2 -Measurement Series mE1&2 mE1&2 mE1&2 mE1&2 mE1&2 mE1&2 -Component - - - - X X -Assetmap X X X X X X -Property pE1 pE1 pE1 pE1 pE1 pE1 pE1Report X X X X X X -Monitoring X X X X - - -

Note: tE1,pE1 and etc. are identifiers for module extensions (see section 3.1.1).

4.1.1. Geothermal plants

Two hundred production holes have been drilled for geothermal plants. Multiple parametersare collected from these holes which provide important information about the developmentand stability of the underlying geothermal reservoir. This data is repeatedly collected henceit is vital that actions such as inserting data into a database and viewing data is easy andautomated so that mistakes are minimized. Geothermal research programs are frequentand therefore it is important to facilitate addition of new methods for processing data.

The estimated number of modules for companies in the geothermal sector are based onthe needs of RE. This data types are:

1. Borholes measurements: Measurements performed at varying depths.

2. Chemical samples: Sampling where each chemical sample has multiple parameters.The samples are analyzed and the results are used for reports.

3. CarbFix data: The pumping down of CO2 into the ground is monitored usingfrequent measurements where each measurement has multiple parameters.

4. SulFix data: Similiar to CarbFix data. The SulFix project aims at fixing hydrogensulfide instead of releasing it into the atmosphere.

5. Tracer measurements: Data is used to view relations between pump-down holesand production holes. Samples are taken and examined and the results are used inreports.

6. Advection data: Periodical measurements for single agents in geothermal water.

7. PT data: Periodical measurements in boreholes for multiple parameters.

8. Geothermal waste water: Sampling from boreholes with parameters for pressureand flow data.

For each data-set listed above there will be an asset group. Each group has a set of modules

22

4.1. Energy market

for data processing. The modules for the data-set groups are listed in table 4.3.

Table 4.3: Number of modules for geothermal data-sets

Modules Data set group Total numberof modules pertype of module

1 2 3 4 5 6 7 8

Property - - - - - - - - 0Time Series - - 1 1 - 6 1 - 9Meas. Series 2 - - - - - - - 2Component - - - - - - - - 0Assetmap - - - - - - - - 0Report - 1 - - 1 - - - 2Sample - 2 1 1 - - - 1 5Monitoring - - 1 1 - - - - 2

Total number of modules per data group 2 3 3 3 1 6 1 1 20

There will be two types of assets for the data: boreholes and power plants. Additionally 8asset groups are defined, one for each data-set.

Basic information for both types of assets (boreholes and power plants) are displayed bythe property module. The power plants will additionally use the asset map module forits components. The components in the asset map are: vapor separator, water extractor,moisture extractor, machine, transformer, condenser and cooling towers. Each of thesecomponents are assets and therefore have their own set of modules. The total number ofmodules are shown in table 4.4.

Table 4.4: Estimate for number of modules for geothermal

Module Number of modules

Property 9Time series 9Measurement series 2Component 0Assetmap 1Report 2Sample 5Monitoring 2

Total 30

The data-sets which Hugfimi has access to are only a subset of the data-sets which REuses. It is likely that RE has more needs which have not been identified and therefore10 modules are added to the estimated number of modules for a geothermal company i.e.total of 40 modules.

23

4. Business model

4.1.2. Hydro power plants

It is reasonable to assume that a system is needed to handle the data for water level,flow and precipitation in a wide area around the power plants. The Icelandic Met Office(IMO) maintains about two hundred sensors which collect these parameters [Ice14c]. Thesesensors are distributed over the country as shown in fig. 4.1.

Figure 4.1: IMO’s water sensor network [Ís12]

An example of collection of data around hydro power plants is a sensor system close toFljótsdalsvirkjun. It contains 7 water level sensors and 17 ground water holes drilled toestimate the environmental impact of the plant [Ice14d]. The time and measurement seriesmodules are useful in this case because the data is measured regularly. The samples fromthe ground holes would be displayed in the sample module and if the data is importantenough to be analyzed and written in a report the report module would be of great use.

Landsvirkjun has a sensor network measuring multiple parameters around their powerplants [Lan14c].

The estimation for the hydro power sector is that they use the same type of modules as thegeothermal sector with the exception of the sample module. The data for the hydro powersector was not as readily available as for the geothermal sector so a modest estimationwas made that they would use 10 modules less than geothermal i.e. 30 modules.

4.1.3. District heating and water facilities

The operations of district heating and water facilities are in many ways very similar togeothermal. Low-temperature boreholes and cold water boreholes are monitored like high-

24

4.1. Energy market

temperature boreholes in geothermal plants. There are multiple parameters collected anddata is also collected from pumping, sewage and purification plants.

The sample module is useful for water samples. The time and measurement series modulescan be used for plotting the flow and temperature data. The asset map and propertymodules are useful for displaying the assets inside the plants and their basic information.The report and monitoring modules are used for automating data processing for reportsand monitoring of the data.

Because of the extent of operations district heating has it is estimated that the numberof modules used are the same as in geothermal i.e. 40.

4.1.4. Power transfer

The power grid is composed of the power transmission network that is solely operated byLandsvirkjun and the power distribution system which is operated by multiple companies.RE and HS-Veitur operate in the South- and Southwest, Norðurorka in Akureyri, Westfjordpower company in the Westfjords and Reyðarfjörður’s Electric Utility in Reyðarfjörður.Finally RARIK distributes electric energy across the country. This section describes howEYK could be used by these companies.

Power transmission:

The power transmission network is composed of both underground and overhead cables.The overhead cables span over 3,000 km and they are supported by masts which arepositioned at 300 meter interval [Man14]. From this it is clear that over 10 thousands ofmasts are in Landsnet’s transmission network alone. There are also 149 substations atLandsnet [Lan14a]. The structures are labeled with a code which is used for identificationof assets in various systems [Lan14b]. This code is similar to the code used at RE in thatit is not easy to remember or read. The accessibility and identification of assets could bemade easier through the map interface of EYK and the asset map can be used for themast and the components module for the mast’s components.

Three types of assets are likely needed for power transmission i.e. a substation, a trans-former station and a transmission mast. All of them have also use for a property andcomponent module and each of their components have a property module. Their compo-nents are:

• Transmission mast: insulator, sphere hooks, suspension clamp, anchor clamp andground connectors.

• Transformer station: cable mounts, safety, washers, circuit breakers and more

• Substation: transmission mast, transformer, current-transformer, lightning rod, iso-lator and circuit breakers.

25

4. Business model

From this use case a conservative estimate is 30 modules for power transmission.

Power distribution:

RARIK’s distribution network is the largest in Iceland spanning over 8,000 km in 2012.About half of the network is overhead cables which means assuming the same distance iskept between each mast like at Landsnet that the total amount of high voltage masts areabout 15 thousand [RAR14].

The distribution network differs mostly from the transmission network as it has a lot morepoints of supply and substations and it also has street cabinets (see fig. 4.2).

Figure 4.2: An example of RARIK’s street cabinet and its content

The Iceland Construction Authority (ICA) demands that all companies which operate inpower transmission or distribution check all their properties every 9 years [Ice14a]. Whenassets are checked their state is assessed and registered into a maintenance system. For eachassessment there can be numerous components inside an asset that need to be assessed likee.g. in a street cabinet the washers, circuit breakers and fuse plugs and more need to beexamined. There have been problems with collecting these assessments and register themat the right place in the systems. EYK can help with this problem by setting up assetswith a component module. Every component would then have a module with connectionto the maintenance system and this way the assessments can be registered on site.

For power transmission and distribution it is estimated that 6 types of modules are used.Time and measurements series modules are set up for voltage and current measurementsand repeated readouts. Components module is used for asset like street cabinets and high-voltage masts and at other places where a simple view over asset components is needed.Assetmap module is used for substations, transformer substations and other large assetswhich include a number of components. The property module shows the basic informationof all assets and their components while the report module is used to ease issuing ofprogress reports.

26

4.2. Sales model

The total number of modules estimated for power distribution is 30.

4.1.5. Monitoring

The National Energy Authority is the sole member in the monitoring sector. It monitorsand authorizes constructions for power plants and has a database which contains allboreholes, water facilities, electric power plants, heating plants and more for public use.In total there are over 7 thousand boreholes and over 500 water facilities, electric powerand heating plants registered into their database [Nat14b]. The data is displayed in theirweb application [Gag14] which was used to estimate the requirements for modules. In itare e.g. boreholes (production and pump down), thermal springs, swimming pools andmore. Many of these assets have additional data such as reports attached to them. It isestimated that at least 10 types of assets are used each with its own property module with“pE1” for attaching reports. The sample module could also be used for the sample datathe receive from RE (and probably more companies). Therefore the total estimation fornumber of modules is 20 modules.

4.1.6. Summary

The Icelandic energy companies, their operations and their use cases of EYK have beendiscussed. The use cases were used to estimate the size of each EYK system a companywill have. It is the size of the system that dictates the cost each company pays for itsversion of EYK. The pricing model is explained in detail in the following section.

4.2. Sales model

The strategy on how EYK would be sold was originally built upon a subscription modelwhere the cost varies depending on the number of modules a customer uses. The modelprovides modest income at first because the number of customers and modules in use arenot many. However as the number of customers increases and as each customer requestsfor more features (more modules) the revenue from subscription can become great.

Early in 2015 a partnership agreement was reached with Mannvit, an international engi-neering consulting firm. With that agreement the original sales model was replaced witha simpler model.

4.2.1. Original sales model

The architecture of EYK is composed of a base, modules and database interfaces (asdescribed in chapter 3). These concepts are the building blocks of the original sales model.

27

4. Business model

The base part of the system is paid for once as part of the initial cost. A fixed pricedis paid for each module and extension used. This cost is intended to meet the costs ofdeveloping a new module or extension. There is also a setup cost for EYK and it mostlyinvolves creating database interfaces for the customer’s databases. The environment atcustomer’s site are different and hence require different amount of time for the developersto setup. To take this variability into account the solution was to use a time rate forthe setup. Finally there is a subscription cost for EYK. It is measured by the size of theinstance of EYK the customer has. The size is estimated by the number of modules used.

4.2.2. Current sales model

The current sales model was created in cooperation with Mannvit. It takes into accountthat EYK currently does not have many modules available and is therefore not ready touse a model built around charging for use of modules. In the current model the customerrequest functionality and Hugfimi estimates the amount of time needed to finish that work.The time estimate with the time rate of the developers results in a cost estimate for theproject.

This model is simply contract work and has the benefits that income revenue is high rightfrom the start assuming that customers are regularly requesting new features.

4.3. Summary

The purpose of this chapter was to establish the Icelandic energy market’s need for EYK.The operations of the energy companies were analyzed and use cases were found for theirassets. A lot of data is collected and processed by these companies and hence there weremany use cases found. From the results obtained from the research it is reasonable toestimate that EYK is a solution that Icelandic energy companies have great use for.

Following the market research the sales model was discussed. The original sales modelwas first described but it uses modules and database interfaces to estimate the price. Thecurrent model was created when Mannvit began partnership with Hugfimi. It relies oncustomers being diligent in requesting new features that they are ready to pay for.

28

5. Technologies

Up to this point the focus has been on the business model of EYK but the rest of thethesis will focus on the software development.

EYK is a web application that has four technologies at its core: a web framework, aweb server, a map software and a database. Those technologies were carefully chosen asthey have great influence on the project. In this chapter the process of selecting thesetechnologies is discussed.

5.1. Web Framework

Once the decision was made to create a web application a web framework was needed. Atthis time there is immense activity in web development and hence there is an enormousrange of web frameworks available. These frameworks vary in scope where some striveto be the complete package while others go a more minimalistic path by offering only aminimum set of features for getting a web application up and running.

It is important for a developer to know what attributes are most important for the projectand use that criteria to filter out the choices. For a web framework to be considered forEYK it needs to meet certain requirements. They are:

1. The framework uses a programming language that the team is familiar with.

2. The framework is a full-stack web framework i.e. is able to handle everything neededto build a complete web application.

3. The framework is open source, well supported and with good documentation.

4. The framework is mature and has a proven track record in production environment.

The vast majority of the programming language experience of both developers is in Python.Therefore using requirement 1 a constraint was set for the framework that it has to usePython.

A list of Python based web framework was found on the official website for the Pythonprogramming language [Iwa14]. The frameworks on the list can be categorized into a“full-stack frameworks” and “non full-stack frameworks” but the former group is only oneto meet requirement 2.

29

5. Technologies

The most popular of the full-stack frameworks is Django [JW14] but web2py [DP15]and TurboGears [Dan15] are two other popular choices. Of these three Django has byfar the most active community and the best documentation and therefore best meetsrequirement 3.

Django was initially released in 2005 and has been used by large companies such asPinterest [JW14], Instagram [Eng14] and Disqus [Rob14]. Django therefore easily meetsrequirement 4 and was chosen as the web framework in EYK.

5.2. Map software

An interactive map will be a prominent feature of EYK and therefore a good tool forcreating one is needed. An interactive can be split into two services. One is the Tile-Map-Service (TMS) which provides tiles for the map. The other one is the Javascript librarythat consumes the tiles, combines them into a complete map along with multiple tools tointeract with it.

The TMS are easily changeable and the option will be made available for customers touse their services and therefore not much research went into choosing a TMS. CurrentlyEYK is using a service from Google for the map tiles.

The choice was between Leaflet [Aga14a] and OpenLayers [OSG14a] for a Javascriptmap library. Both libraries were tested and proved to have its pros and cons. They bothprovide the necessary functionality for an interactive map in EYK such as: panning,zooming, pinning markers and supporting TMS. OpenLayers has a larger community anda wider set of features but Leaflet was much easier to use. Leaflet was the clear winnerfor both developers but there were concerns on if it had not been tested enough in aproduction environment. Those concerns did not last long as research on where Leafletis used in production showed that large services such as “Já.is”[Já14], Foursquare andCitymaps [Aga14b] are using it with success. In fig. 5.1 is a good example on how “Já.is”uses Leaflet for its product.

30

5.3. Database

Figure 5.1: A map interface for “Já.is” with many features.

5.3. Database

Django officially supports Relational database management systems (RDBMSs) [Fou14a]and there are non-official branches of Django which support Not-Only-SQL (NoSQL)databases. There are base requirements that a database needs to meet to be a viablechoice for EYK. The most important ones are:

1. The database is supported by Django.

2. The database has good documentation and a well documented API.

3. It is preferable that the database is open source.

The databases PostgreSQL [Pos14], MySQL [Ora14] and SQLite [Hip14] are officiallysupported by Django. SQLite is not recommended for production environment but servesas a useful test database as it can be run in-memory and therefore making tests runfaster. MySQL is developed by Oracle Corporation and is released under two licenses. Thelicense intended for commercial usage is not open source hence it fails criteria 3. On theother hand PostgreSQL is open source and the developers of EYK have used it before.It has a spatial database extension, PostGIS [OSG14b], which Django supports throughGeoDjango [Fou14b]. The extension offers specific fields for Django models that are relatedto spatial objects e.g. a line, polygon and etc.

There was an interest to test NoSQL databases, specifically MongoDB [Inc14]. It meetsthe requirements of EYK except it is not officially supported in Django. No official support

31

5. Technologies

is a deal breaker in this case hence PostgreSQL is the database for EYK.

5.4. Web Server

The market for web servers is dominated by the open source solution Apache [Ltd14] withover 50% market share of active sites with the closest competitor, nginx [ngi14], at 15%market share. These two of the most popular solutions were considered for EYK.

Apache and nginx differ in how they handle processes. Apache creates processes andthreads to handle additional connections and is limited to the amount of memory available.nginx has a pre-defined number of single threaded processes which can handle thousands ofconcurrent connections asynchronously. It is built to handle large numbers of connectionsin a much smaller memory space. As EYK is a decentralized solution it is highly unlikelythat the number of concurrent connections will be in the thousands. Scalability is not alarge concern for EYK.

As Apache is the most popular web server it is naturally also the most documented one.This is an important advantage Apache has over nginx as it becomes much easier to findsolution or help when customizing or fixing the web server. Other good reasons for choosingApache were that Django offers good documentation for it and both developers have usedit before. There were many advantages that Apache had and it was therefore a clear choiceas a web server for EYK.

5.5. Summary

Four core technologies used for EYK were described in this chapter and their selectionwas justified. Django was selected as a web framework because it uses Python and is amature and well documented project. For the map software, two options were researchedfor building interactive maps: Leaflet and OpenLayers. The former proved to be very easyto use, has been used successfully in a production environment and offers the necessaryfunctionality for EYK. PostgreSQL was picked as a database as the developers had experi-ence in using it, it is free and it has official support in Django. Finally Apache was chosenas a web server and the main reason for choosing it was that it is a tested and proven webserver that majority of active sites on the web run on.

32

6. Development processes

This chapter describes in three parts how the software development is organized. Thefirst part discusses project management that was built on Scrum. The second focuses onrevision control which includes a branch model and the versioning of EYK’s code. Finallythe third part discusses configuration management in general where the various processesfor maintaining and updating EYK’s servers are automated.

6.1. Project Management

In the beginning of the project and before actual development started work flows werecreated to organize the work. These work flows are built on Scrum [SS14] but have fewmodifications to accommodate the size of the team which is smaller than a normal Scrumteam usually is.

6.1.1. Tools

Scrum

The Scrum framework [SS14] consists of a Scrum team and their associated roles, eventsand artifacts. Normally there are three roles: product owner, Scrum master and thedevelopment team. The product owner is responsible for ensuring that the team deliversvalue to the business and acts as a communication link between the team and the client.The Scrum master is responsible for ensuring that Scrum processes are used as intendedand handles any distractions which could impend the team’s work. Finally the developmentteam is responsible for delivering a potentially shippable increment of a product at theend of each sprint. The product owner can be part of this team.

The main artifacts in Scrum are the product and sprint backlog. The product backlog is alist containing all the requirements for the product which have not been completed. Thesprint backlog is then simply all the work which the development team must finish for agiven sprint.

The sprint is the main event in Scrum. It is a time period of up to four weeks in whichthe Scrum team tries to finish a goal which often involves delivering code into production.

33


In each sprint are three type of events. Daily Scrum are short daily meetings where teammembers describe their work for the day before and the current day and the impedimentthat might prevent their work. Sprint review and retrospective are then held at the endof each sprint. In sprint review the work is reviewed and the product backlog is updatedaccordingly. In retrospective the team members have the opportunity to reflect on theirwork and make some improvements on their work flow. Normally story points are used toestimate the size of a task. A story point is a relative size e.g. one task can be evaluated byone team member as four story points but only two story points by another. This simplymeans that the latter one could be more experienced and therefore needs less time tofinish the task.

Trello

Trello [Tre14] is a free web-based project management application. The team had usedTrello before for non-software development purposes but wanted to try it for managingthe development of EYK. The application is very simple and flexible in use. Tasks areorganized using three components: boards, lists and cards. A board is at the highestorganization level and contains an arbitrary number of lists and cards. A list then containsan arbitrary number of cards and then each card has a range of methods to present datae.g. text can be written in a rich text format and images can be attached to each card.

Trello was a simple choice to make in the beginning but it did not scale very well withthe project. The number of boards and tasks grew and it became difficult to get a goodoverview of the sprints and tasks. Also Trello did not have any options for using metricsto measure the development. For these reasons it was eventually replaced.

Jira & plugins

Jira [Inc15] is a issue tracker software that provides bug tracking, issue tracking and projectmanagement functions. It is the current project management tool for EYK and has a widerange of features and plugins available. It offers support for Agile development wheresprints and tasks can be visualized as boards and cards similar to Trello. It also providesmany metrics for analyzing the state of the project. An important feature in Jira is thatit has a plugin for time tracking which became a requirement for the team once the newsales model (see section 4.2.2) had been set.

6.1.2. The work flows at Hugfimi

The work flow at Hugfimi is originally built on Scrum but is different in many ways mainlybecause the size of team is not large enough to fulfill all of Scrum’s roles. There is noScrum Master nor Product Owner but developers work close together and keep each otherworks in sync.

34


For most of the project sprints have been used and they were 1 week long. In the beginningof the sprint a meeting was held where the previous sprint was discussed and the next oneplanned. The review of the last sprint focused on evaluating the actual time spent on eachtask versus the story points given to the task. This was done in order to achieve a betterestimate for future tasks. There were no formal daily meetings but the team memberscommunicated frequently each day about their work and were therefore in good sync.

Sprints are no longer used at Hugfimi. The team was never able to have a good estimate ofstory points for tasks and therefore the useful information that would normally come froma sprint were meaningless. The main reason for this problem was that neither developercould restrict their work to these tasks because new tasks that had to be dealt with assoon as possible would regularly pop up in the middle of a sprint. These tasks could rangefrom being a bug in the system or a business meeting which both team members hadto attend. Sprints were abandoned as these circumstances were not going to change andtherefore neither developer has the luxury of being able to focus solely on the tasks in asprint.

Instead of sprints the team now has epics. An epic is a large task which represents afeature that a customer has requested. Each epic has a time and cost estimate that thecustomer has agreed upon. An epic is broken into smaller tasks which the team workson. In order to evaluate the progression of an epic the time tracking tool in Jira is usedto compare the time spent on an epic to the original time estimate. As an epic does notrequire a fixed end date the team has the flexibility it needs to handle unexpected taskswithout much hassle.

A product backlog is still used although with some differences. When sprints were usedit contained tasks that were not fully constructed and had not been assigned to a sprint.In Trello those tasks were organized with lists that indicated the task type (see fig. 6.1a).Now as the sprints have been abandoned the product backlog only contains tasks that arenot fully constructed and in Jira each task has a field for its type and therefore multiplelists are not needed as in Trello. The product backlog in Jira therefore consists of onelarge list as shown in fig. 6.1b.

35


(a) The product backlog in Trello.

(b) The product backlog in Jira.

Figure 6.1: Product backlogs in Trello and Jira

A separate backlog had to be created for each sprint before when sprint were still used. InTrello that meant a new board had to be created for each new sprint. These boards werethen seperated into six lists as shown in fig. 6.2.

36


Figure 6.2: A sprint backlog in Trello.

The task types that belong in each list are as follows:

Sprint backlog: Tasks that work have not been started on.

In progress: Tasks that are being worked upon.

Test bug reports: Tasks that did not pass tests and need to be fixed.

Ready for tests Tasks that have been implemented and are waiting to be tested.

In testing: Tasks that are being tested.

Done/Live: Tasks that are completed.

The flow of tasks through these lists is shown in fig. 6.3 where each state represent oneof the list from fig. 6.2. A task in a sprint begins in the sprint backlog until a developerstarts working on it. When the developer has implemented the task he can either starttesting it or go work on another task. Either way the task will eventually be tested andif it passes it is marked completed however if it fails then it needs to be re-implementedbefore being tested again.

The team was satisfied with this work flow early in the project as it was eager to add newfeatures to the system in order to make EYK more attractable to customers. This meantthat most of the time a task was left waiting for testing once it had been implemented.The team expected that once EYK had enough features to attract new customers thenmore time would be spent on testing.

Now instead of sprint backlogs there exists one board that contains all the tasks thatare ready to be worked on. Other tasks are kept in the product backlog until they have

37


Figure 6.3: The sprint work flow for the first iteration

been fully constructed. In order to have enough tasks to work upon the product backlogis regularly visited by the team and tasks are constructed and moved from the productbacklog and into the board. The board is shown in fig. 6.4 and all tasks go to the leftmostcolumn, i.e. “TODO”, receives tasks from the once the are fully constructed.

Figure 6.4: A JIRA board.

Today more emphasis is put on testing than in the beginning of the project howeverimprovements can still be made. There is one notable addition to the work flow which isthat all new commits go through a review process by the other developer to get acceptedinto the code. This has had led to the developers being in better sync with each otherwork.

38

6.2. Source control

6.2. Source control

The source code is revisioned with a revision control system and a branching model wascreated to organize how code is added to EYK. A simple versioning system was alsocreated to keep track of the changes that are made to the system.

6.2.1. Tools

Git

Git [Sof14] is a distributed revision control system. There are other options for revisioningcode however the team has used Git before and are very satisfied with its performanceand therefore it was an easy decision to choose Git.

6.2.2. Branching model

In the beginning of the project the team did not use a branching model but when they had abetter picture of EYK’s architecture they created one. The model is built on gitflow [Dri14]but has minor modifications. It categorizes branches into five types:

Master One of two branches which are used to record the history of the project. It storesthe official release history. It does that because every commit to it only containscode which is production ready. Tags are then used on the branch to keep track ofthe software releases.

Hotfix The only branch which forks directly of master. It is used to quickly fix bugs in theproduction code. The commits from this branch merge into master and developmentand when that is done it is removed.

Release A branch used to prepare code for production environment. Branches off devel-opment and is merged into master and development and then removed. Bugfixesfound on release branch can be continuously merged into development branch. Onthis branch the commit history is examined in order to recognize the correct versionnumbers for EYK.

Development The other branch which records the history of the project. Contains thedevelopment history (complete history) of the project.

Feature Whenever a new feature is introduced a feature branch is forked of development.When a feature is completed it is merged into development and the feature branchis removed. Feature branches are categorized into the following types:

Module Changes on a module. If the changes involve adding new features (exten-

39


sions) then it increments the minor version number. Otherwise if it is main-tenance on existing functionality then it increments the maintenance versionnumber.

Base Changes to the base of EYK. If the changes involve adding new features (ex-tensions) then it increments the minor version number for EYK. Otherwise if itis maintenance on existing functionality then it increments EYK’s maintenanceversion number.

The model is still used by the team and they are very please with it. In fig. 6.5 is agraphical illustration of the branching model.

Figure 6.5: Git flow branching model

6.2.3. Versioning

The current version of EYK is 0.8 and the team is developing towards releasing version1.0. There was not much thought put into the meaning of these version numbers but theteam decided that version 1.0 would be the first production version of EYK. A morestructured system for the versioning has been created and it will be from version 1.0. Ithas three integers where the first number stands for major version number, the second forminor version number and the third for maintenance version number. These numbers areincremented for the following conditions:

1. (Major) Changes that have a fundamental change in the architecture of EYK. Minoror maintenance updates following a major update are not required to be compatiblewith a previous major version.

2. (Minor) Incremented when large features are added to the system. A common large

40

6.3. Configuration management

feature is a new module and an extension to a module. A large extension to the basepart of the system is also considered a minor update.

3. (Maintenance) Incremented when fixes, enhancements and small features are addedto the system.


Maintaining a server can be a tedious task. Maintaining multiple servers with differentconfigurations can easily bring the pace of development to a halt but that can be preventedby automation. To simplify the process of automating a tool like Ansible will be used. Itprovides a simple way to automate tasks such as deployment and as a demonstration theautomation of the deployment process to EYK’s demo server is described in section 6.3.3.

6.3.1. Tools

Vagrant

In order to minimize the differences between the production and development environmenta virtual machine. Vagrant [HB14] is a tool for managing Virtual Machines (VMs) viacommand line interface. It has a large community that provides a wide range of Vagrantboxes. These boxes are pre-built VMs and can include various packages. The configurationsfor a Vagrant box are all defined in one file, Vagrantfile. The configurations for EYK areshown in listing 6.1 and listing 6.2.

13 c on f i g .vm. box = "CentOS -6.4"1415 c on f i g .vm. d e f i n e c on f i g . user . hostname , primary: true do | host |16 host .vm. hostname = con f i g . user . hostname17 end1819 c on f i g .vm. box_url = "http :// developer.nrel.gov/downloads/vagrant -boxes/CentOS -6.4-

x86_64 -v20131103.box"

Listing 6.1: Vagrantfile: Configurations for the virtualbox

In listing 6.1 a virtual box is chosen and the URL to the same box is given in case if ithas not been downloaded onto the machine before.

25 c on f i g .vm. network :forwarded_port , guest : 8000 , host : 4567 , auto_correct :true

Listing 6.2: Vagrantfile: Configurations for networking

In listing 6.2 port 8000 on the virtual machine is chosen to be forwarded on port 4567

41


on the host machine so when EYK is running locally it is accessible in browsers throughlocalhost:4567.

Finally as CentOS was chosen as the operating system for EYK, a Vagrant box with itwas fetched from Vagrant’s cloud service. The configuration for the OS are then definedin a Vagrantfile that is stored in the EYK repository so in case of a corrupt system thebox and its configurations are easily replicated.

Ansible

There are two ways to automatically have a virtual machine configured. One is to builda Vagrant box with the correct setup but that would require a new box to be built eachtime changes are made to the OS e.g. when a new system package is installed. This isinefficient as it can take a long time to build a box. Another way is to use a configurationmanagement system that can handle all configurations on the servers and computers thatEYK is run on.

The team had never used a configurations management system before and therefore Pup-pet [Lab14] the most popular one was tested. It has plenty of features and provides botha CLI and a GUI interface but it has a steep learning curve. It also uses a pull basedmechanism which means that Puppet must be installed on each server running EYK. Theteam found wanted to look further for a simpler solution.

Ansible [Ans14] seemed to be a simpler solution and was therefore chosen next. It uses apush based architecture instead of a pull based which means that a machine with Ansibleinstalled can push configurations on other machines through SSH regardless if they haveAnsible. In fact the only requirements are SSH and Python both which are available bydefault on the servers. After using Ansible for few weeks it proved to be easy to use andhas been used successfully to configure a server running EYK. The conclusion was thatAnsible was a good fit for the project and is being used to handle various tasks at Hugfimi.

An important feature of Ansible is that it is idempotent meaning that changes are notmade to the system unless needed. An example would be a task that installs a packagewhich is already installed on the system. Ansible acknowledges that it is installed and skipsthe task instead of re-installing. Ansible idempotency also makes it possible for developersto quickly assess if a problem in EYK is because of wrong configurations in the OS andnot in EYK’s code.

In section 6.3.3 a demonstration is shown in how Ansible is used to deploy EYK to a serverbut further information can also be found in about Ansible can be found in appendix C.

6.3.2. Deployment

Software deployment is the process of making software system available for users. Thecomplexity of the process depends on the software and its environment. In some cases

42


systems need to be deployed to multiple servers, each with its own configuration, while inother cases the system is deployed to one server. The frequency of deployments is also animportant factor because as the frequency increases, the greater the need for automation.

EYK needs to be deployed to multiple servers. These servers are located at customers siteand can take some work to connect to as in RE’s case.

The customer can choose a subset of features from EYK resulting in different systemsbetween customers. The automation has to incorporate actions which ensure that onlythese subsets of features are enabled. The frequency of deployment is not high but insituations where fixes are made to the system it is necessary that the process goes fastand efficiently.

In the following section a part of the deployment process of EYK’s demo is described.The server is hosted on a cloud service and is running the same operating system as otherproduction servers. This means that a large set of tasks are similar between the demo andother servers and can therefore be reused.

6.3.3. Example: Deploying to demo server

An example of the Ansible project layout for the demo server is shown in listing 6.3. Itcontains host files, playbooks and variable files. Some of these concepts are specific forAnsible and will be explained in the following text.

� �1 . # top level of an ansible instance2 |-- demo.yml # main playbook for the demo group3 |-- group_vars # group specific variables4 |-- hosts # hosts and host groups ~(ip addresses , user names etc.)5 |-- roles # a level of abstraction for organizing playbooks6 |-- django7 |-- files # files that need to be transferred to or run on the hosts8 |-- handlers # tasks that are run on events9 |-- meta # contains files that establish role dependencies10 |-- tasks # all tasks a playbook runs.11 |-- templates # variables to substitute information during creation12 |-- vars # variables for roles , can be used in configuration files13 |-- site.yml # main playbook for Ansible , contains all active plays� �

Listing 6.3: Ansible project layout

To run this project, one simple command is needed:

ansible-playbook -i hosts site.yml

In the command are two parameters. The first one is indicated with an -i flag and is forthe inventory file, i.e. hosts. The hosts file contains the addresses to the systems whichAnsible maintains. Listing 6.4 shows the content of the hosts file for EYK. It assigns theip-address to the demo server to the group name demo and also provides the ip-addressunder the variable ipaddress. This variable is accessible in all the tasks that are run onthe demo server.

43


1 [ demo ]2 188 . 226 . 254 . 155 ipaddre s s =188.226 .254 .155

Listing 6.4: Ansible host file

The second argument in the command above is for the project playbook site.yml. Aplaybook in Ansible is where the design of the configuration management is set. It mapshosts to Ansible roles and sets the execution order of those tasks. In the case for EYK eachserver has its own playbook and they are included in the project playbook (see listing 6.5).

1 ---2 # Main playbook3 - i n c lude : or . yml4 - i n c lude : demo . yml

Listing 6.5: The main playbook

Listing 6.6 shows the playbook for the demo group and the roles that have been set torun on the demo server. From line 3 to 6 the role common is defined to be run by a rootuser on the demo server. The root user is needed to set up permissions for the vagrantwhich the rest of the roles (lines 8-14) are run under.

1 ---2 # Playbooks for EYK demos3 − host s : demo4 user : {{ roo tu s e r }}5 r o l e s :6 - common78 − host s : demo9 user : vagrant10 r o l e s :11 - apache12 - po s tg r e s13 - django14 - demo

Listing 6.6: The roles used for configuring the demo server

For cases when multiple hosts use the same playbooks then group variables can be usedfor the different values between the hosts. Group variables for the demo group are in thegroup_vars directory (see listing 6.3) and its content is shown in listing 6.7. The rootuservariable is defined because the root user can be different between the EYK servers. Fromline 5 to 15 the subset of features that are enabled for the demo are defined. By using thisapproach the set of features each customer has can be easily maintained. Finally from line17 to 21 the configurations are defined for EYK’s database.

44


1 ---2 # Variables specific for DEMO3 r oo tu s e r : root45 db_inte r face s :6 - db_inte r face s . db_or . bo r eho l e s7 - db_inte r face s . db_or . a s s e tda ta8 - db_inte r face s . db_or . a i r q u a l i t y910 #base_extensions:1112 modules:13 - modules . u t i l s14 - modules . p l o t15 - modules . p r op e r t i e s1617 database:18 name: y ggd r a s i l19 user : apo l l o20 al lowed_hosts :21 - hugf imi . com

Listing 6.7: The group variables for the demo server

Roles are used to group content. A directory layout for a role is shown in listing 6.3where each directory has a special purpose. In the tasks directory are the playbook(s) thatcontain the tasks that are run at user request while handlers contains tasks that are eventdriven. In the files are files that are transferred to the remote system. In the templatesdirectory are files which are also transferred to the remote system. What separates themhowever from files in the files directory is that they are templates which contains variousplaceholders that are rendered when they are transferred. This is very useful as it allowsthe same template file to be used for different hosts by using different variables for eachgroup. In the meta directory dependencies can be declared for other roles and in the varsdirectory are all the variables specific for the role.

In this example only one part of the Django role will be described but a full example canbe found in appendix D.

Role: Django

At the point when the Django role is run the webserver and database have been installedand configured for the environment and therefore EYK’s repository can be fetched andconfigured. The role django is intended for task which act on EYK’s code. There are manytasks involved and therefore they are split into multiple files like shown in listing 6.8.The tasks in the git “sub-role” are described below whereas the other are described inappendix D.

45


1 # Django tasks2 ---3 - i n c lude : g i t . yml4 - i n c lude : yum. yml5 - i n c lude : pip . yml6 - i n c lude : database . yml7 - i n c lude : django . yml8 - i n c lude : common . yml

Listing 6.8: List of subroles for the Django role

Before any actions can be made on the code it first needs to be pulled to the server. As afirst task in listing 6.9 the directory for EYK is created and then the head of the masterbranch is pulled.

1 ---2 − name: System | Permiss ions | c r e a t e / vagrant / s r c d i r e c t o r y3 remote_user: "{{ rootuser }}"4 sudo: yes5 f i l e : des t=/vagrant / s r c6 owner=vagrant7 group=vagrant8 s t a t e=d i r e c t o r y9 tags :10 - product ion11 - development12 - c l one1314 − name: Django | Git | c l one EYK repo s i t o r y15 sudo: no16 g i t : repo=git@github . com:hgy l f a son /eyk . g i t17 dest=/vagrant / s r c18 accept_hostkey=yes19 update=no20 ve r s i on=master21 tags :22 - product ion23 - EYK24 - c l one25 no t i f y :26 - Django | S e t t i n g s | update s e t t i n g s f i l e

Listing 6.9: Ansible tasks for the git part in the Django role

After the pull, a handler (listing 6.10) is activated which renders the settings template (list-ing 6.11) with the variables shown in listing 6.7. For deployment to customer’s server thisis used to activate the features which they have purchased.

1 ---2 − name: Django | S e t t i n g s | update s e t t i n g s f i l e3 remote_user: "{{ rootuser }}"4 sudo: yes5 template : s r c=r o l e s /django/ f i l e s / product ion . py . j 26 dest=/vagrant / s r c /eyk/eyk/ s e t t i n g s / product ion . py7 owner=vagrant8 group=use r s

Listing 6.10: Ansible handler for the git part in the Django role

46

6.4. Summary

1 # coding: utf -823 from . base import ∗45 DEBUG = False6 TEMPLATE_DEBUG = DEBUG78 ALLOWED_HOSTS = [9 {% f o r host in database . a l lowed_hosts %}10 u’{{ host }}’ ,11 {% endfor %}12 ]1314 DATABASES = {15 ’default ’: {16 ’ENGINE ’: ’django.db.backends.postgresql_psycopg2 ’ ,17 ’NAME’: ’{{ database.name }}’ ,18 ’USER’: ’{{ database.user }}’ ,19 ’PASSWORD ’: ’{{ database.password }}’ ,20 ’HOST’: ’{{ database.host | default(’’) }}’ ,21 ’PORT’: ’{{ database.port | default(’’) }}’ ,22 }23 }242526 INSTALLED APPS += (27 # installed database interfaces28 {% f o r i n t e r f a c e in db_inte r face s %}29 ’{{ interface }}’ ,30 {% endfor %}3132 # installed base extensions33 {% f o r ex tens i on in base_extens ions | d e f au l t ( ’’ ) %}34 ’{{ extension }}’ ,35 {% endfor %}3637 # installed modules38 {% f o r module in modules %}39 ’{{ module }}’ ,40 {% endfor %}41 )

Listing 6.11: Ansible template for Django settings in production environment

At this point all Git related tasks have been completed and the Django specific configura-tions for the database, webserver and other components of the system can be set.

6.4. Summary

The work flows used at Hugfimi were originally built on Scrum but have since deviatedfrom it. Sprints are no longer used although meetings are held regularly. Early in theproject not much emphasis was put on testing but as the project moved along more focushas been put into it.

The code is revisioned with Git and a branching model was created to organize how codeis added to EYK. Also a simple versioning system was created to keep track of the changesto EYK.

47


Each customer will have his own EYK server and therefore careful considerations weremade to ensure that the process of maintaining them would not become too time consuming.The server hardware is not the responsibility of Hugfimi however the state of the OS is.Ansible enables the developers to automate the task of checking and setting up the servers.

48

7. Testing

All development processes have been described up to this point except for testing processes.In the beginning of the project there was not much emphasis on testing. As has beenmentioned earlier the developers often implemented a task and left it waiting for testing.In the worst cases the code was tested many weeks after its implementation. One of thesecases is described later in the chapter.

Testing is done on three levels: unit, integration and system testing [ABDM01]. Unit andintegration tests were created together and the test suites typically contain a mixtureof them. System testing is executed with a different tool and is done after unit andintegration testing are completed. In addition to aforementioned levels, the operatingsystem environment that EYK operates in is automatically checked by Ansible. Ansibledoes this by asserting that the state on the servers corresponds to the state defined inAnsible playbooks. If there is a difference then Ansible either corrects that state or alertsthat the server is in an incorrect state. This functionality becomes very useful in testingas it enables developers to quickly check if a problem with EYK is because of bad code oran incorrect state in the OS.

This chapter begins on discussing the tools used for testing. Following that the testinglevels are discussed which includes examples. Then future improvements are introducedwhich mainly include to adopt a test driven approach.

7.1. Tools

unittest

unittest [Pyt14], formerly known as PyUnit, is the standard unit testing framework forPython. It is a xUnit [Fow14] framework meaning it has the same main components asother well known xUnit frameworks such as JUnit (Java). It has test case as its smallestunit of testing and a collection of them are test suites. The test suite is used to aggregatetest cases or other test suites which should be executed together. It supports the use offixtures to setup up a known state for the tests which is necessary to isolate the unit beingtested from the rest of the system. The test execution is broken into three parts. First isthe setup which usually involves preparing fixtures, then it is the test itself and finally itends in tear down where the environment is cleaned up for other tests which could follow.The test runner is one of the main components and handles orchestrating the executionand providing a readable outcome to the user.

49

7. Testing

Model mommy

In Django, a separate blank database (namely, test database) is created whenever codeis tested that uses Django models. For these tests to work, the test database needs to bepopulated with data for the Django models that are being tested. Model mommy [dS14]enables users to easily create these fixtures. One feature of Model mommy is that it autogenerates data for undefined fields. This is useful as all fields must be defined but eachtest is usually however not interested in all the fields at once. Model mommy also handlesrelations between models so the user only needs to define fields for one model and Modelmommy auto generates fixtures for all related models.

Model fields that are important for a test are usually defined manually as their valuesneed to be known in order to predict the correct outcome of the tested code. These fieldscan either be defined at the same time the fixture is instantiated or in recipes. A recipe issimply a Python file that contains Django model fixtures but their advantage is that theycan be reused.

QUnit

QUnit [JQu14] is a Javascript unit testing framework which shares many characteristicswith a xUnit framework like unittest. This similarity makes it easier for developers toswitch between them. QUnit is run through the browser and the test results are presentedon a HTML page.

SinonJS

A common problem in testing is handling of date sensitive code. In Javascript this appliesto the Date object which value depends on the time when it is called. SinonJS [Joh14]is a Javascript library which can be added to the QUnit framework. It provides a clockobject which is used to control Date objects. The tool is under active development andthe code is open source so there is always the possibility to fork the repository and extendthe functionality if needed.

CasperJS

CasperJS [Per14] is an open source Javascript library that offers navigation and testingutilities for a Web-based UI on top of PhantomJS [Hid14]. It is used on system testinglevel to emulate user actions in EYK and test the results of those actions. It is also usedto detect regressions in the appearance of the GUI. It does so by taking screen-shots ofthe elements in the GUI and compare it to a reference image.

50

7.2. Unit and integration testing

coverage.py

Code coverage is used as a guideline for test completion in EYK. QUnit provides coveragebut unittest does not and therefore coverage.py [Bat14] is used for measuring code coveragefor tests on Python code. Coverage.py is run on a CLI and can either output the coverageresults in the terminal or as a webpage.


The tests were created when most of the system had been implemented. The approachwas to create unit and integration tests before moving onto system testing. The first unittests were created for the Python scripts which did not depend on Django as they weremore isolated e.g. functions for converting date and timestamps.

For testing Django files it was necessary to mock the data which the Django modelsused. Model mommy supports the mocking and as an example the asset, assettype andorganization models are used. The models are defined as shown in listing 7.1. Each hasits own class and each field stands for a column in a database table.

1 class AssetType(models.Model):2 name = models.CharField(max_length =255, unique=True)3 plural = models.CharField(max_length =255)4 icon_path = models.CharField(max_length =255)56 def __unicode__(self):7 return self.name89 class Organization(models.Model):10 name = models.CharField(max_length =255, unique=True)11 icon_path = models.CharField(max_length =255, blank=True)1213 def __unicode__(self):14 return self.name1516 class Asset(models.Model):17 name = models.CharField(max_length =255)18 type = models.ForeignKey(AssetType)19 sub_types = models.ManyToManyField(SubType)20 organization = models.ForeignKey(Organization)21 longitude = models.FloatField ()22 latitude = models.FloatField ()23 legacy_id = models.CharField(max_length =255, blank=True)2425 def __unicode__(self):26 return self.name

Listing 7.1: The Django models for assets

The mocks for these models are shown in listing 7.2. Not all of the model fields aredefined for the mocks but they can be added when the mocks are instantiated otherwiseModel mommy automatically creates them with values appropriate for the type of fieldse.g. a random floating value for FloatField. In some cases it is not an option to letModel mommy generate a field e.g. the value for the icon_path has to be an existingpath otherwise an error occurs. In other cases like for latitude and longitude it is not

51

7. Testing

necessary to manually define the value as it is only constrained to the type of field whichModel mommy recognizes. Foreign keys are handled in Model mommy by assigning a mockto the field e.g. the type field in asset mock points to the assettype mock. For a fieldwhich is a foreign key i.e. a relation to another model then it is enough to assign the mockfor that model to that field as is done for the field type in the asset mock.

1 #A recipe for AssetType Model.2 asset_type = Recipe(AssetType ,3 name = u’Borhola ’,4 plural = u’Borholur ’,5 icon_path = u’map/images/Borholur.png’6 )78 #A recipe for Organization Model.9 organization = Recipe(Organization ,10 name = u’Orkuveita Reykjavikur ’,11 icon_path = u’map/images/Orkuveita Reykjavikur.png’12 )1314 #A recipe for Asset Model.15 asset = Recipe(Asset ,16 id = 8,17 name = u’HE -03’,18 type = foreign_key(asset_type),19 organization = foreign_key(organization)20 )

Listing 7.2: Mocks for Django models

All the values which are defined in listing 7.2 are default values and can be overwrittenwhen an instance of the mocks is created. In listing 7.3 a mock object for an asset iscreated by giving the path to the recipes mentioned in listing 7.2 as an argument. Andwith that the assettype and organization mock objects are also created as they haverelations to asset.

1 self.asset = mommy.make_recipe(’oats.asset ’)

Listing 7.3: Using mocks in testing

Model mommy is intended for Django models so when mocks were made for other objectse.g. normal Python classes the approach was to create fixtures in the test suite. In twocases the mock data was stored in a text file because the code involved was reading datafrom those kind of files.

The Django models are used in the Django views. They take a request as a parameterwhich has to be mocked in order to isolate the view. Django provides a test client for thattask. In listing 7.4 the default view for an asset is tested using that client.

For setting up a test, a mock object is created of an asset (line 7), and a false id of anasset is defined (line 9). Then the URL is retrieved (line 17) by using the reverse functionfrom Django but it returns an absolute path reference (a URL without the domain name)when given the name of the URL. Doing this instead of hard coding the URL preventsthe tests from breaking when changes are made in the URL structure. Next the test client

52


1 class OatViewTestCase(TestCase):2 """3 Tests that all views for current app return a 200 status4 """5 def setUp(self):6 # Create necessary data which is common for all tests of type OatTestCase7 self.asset = mommy.make_recipe(’oats.asset ’)8 self.assettype = oats.models.AssetType.objects.get(id=self.asset.type_id)9 self.asset_id_false = 0101112 class AssetDefaultViewTestCase(OatViewTestCase):13 def setUp(self):14 super(AssetDefaultViewTestCase , self).setUp()1516 def test_if_view_returns(self):17 path = reverse(’oats:asset_default ’, kwargs ={’asset_pk ’: self.asset.id})18 client = Client ()19 response = client.get(path)2021 self.assertEqual(response.status_code , 200)2223 def test_if_view_returns_404_on_false_asset(self):24 path = reverse(’oats:asset_default ’, kwargs ={’asset_pk ’: self.asset_id_false })25 client = Client ()26 response = client.get(path)2728 self.assertEqual(response.status_code , 404)

Listing 7.4: Testing a Django view

is instantiated (line 18) and used to retrieve the page for the asset (line 19). Then the testasserts that a page was retrieved (line 21). The latter test differs in that a false asset id isgiven (24) and the test then asserts that the correct error was returned (28).

The test client offers basic functionality to test the views. It tests GET and POST requestsand observes the response. However in system testing another test client is used, CasperJS.The testing in CasperJS differs as it is aimed to emulate user interaction by testing therendered HTML and behavior of the page which includes Javascript.

No unit tests were made specifically to test the code used from third party libraries asthat is done by the maintainers of the corresponding projects. However the effects thecode has in EYK are mostly covered with other unit tests and also by integration tests.

Branch coverage was used as a guideline for the test completion criteria. Sufficient branchcoverage varies between the components of EYK as some functionality is more criticalthan others.

The unit and integration testing went well however as testing was not done on a regularbasis and in parallel to programming then flaws in the programming were allowed to growwith the system. This was most evident in the testing for the datebar, a GUI componentfor time series graphs, and therefore in the following section its testing is discussed.

53

7. Testing

7.2.1. Test case: datebar

The date_bar provides an interface for configuring the range of the data used in graphs.The GUI is shown in fig. 7.1 and it consists of set of buttons and input fields for settingthe time range and a plot button to update the graph.

Figure 7.1: The GUI of the datebar

The complications in testing the datebar are described in the following sections.

The datebar The date_bar consists of many methods and variables. To prevent thatthe names of these objects clash with other objects in the Global Namespace, a feature ofJavascript is used called closures. A closure is an object which combines a function/variableand the environment in which the function/variable was created like a class does in class-based programming languages. As can be seen in line 12 of listing 7.5 the variable DateBaris assigned the closure which the datebar operates in.

12 var DateBar = function(module_namespace){13 var dateRangeId = "#"+ module_namespace +"DateBarDateRange";14 var dpFromId = "#"+ module_namespace +"DateBarFrom";15 var dpToId = "#"+ module_namespace +"DateBarTo";16 var actionBtnId = "#"+ module_namespace +"DateBarActionButton";

Listing 7.5: Datebar closure created

Because the datebar can be used in multiple modules under the same asset it is nec-essary to prepend the value of the variables which are rendered in the HTML withmodule_namespace to prevent name clashes. These variables are private variables inthe datebar closure because they are prefixed with the keyword var and are therefore notaccessible outside it.

54


51 var disableAction = function (){52 $(actionBtnId).prop(’disabled ’, true);53 };5455 var enableAction = function (){56 $(actionBtnId).prop(’disabled ’, false);57 };58 var setRangeLast = function(days) {59 db.endDate = new Date();60 db.startDate = dateBefore(db.endDate , days);61 setDate($(dpFromId), db.startDate);62 setDate($(dpToId), db.endDate);63 };64 var onClick = function () {65 disableAction ();66 db.onClickFunction(67 getDateFrom (),68 getDateTo (),69 enableAction70 );71 };

Listing 7.6: Closures nested in the datebar

Most of the datebar closure consists of other nested closures. Some of them are bound toevents e.g. onClick, and use the variable db widely e.g. in lines 59, 60 and 66 in listing 7.6.To understand the use of this variable a little explanation of the keyword this is needed.It refers to the object which the function is a part of. For functions in the datebar closurethe keyword refers to the DateBar variable but inside the nested closures the keywordrefers no longer to that variable but to the variable which that closure is assigned to. Forthis reason the db variable was created to store the reference to the datebar closure.

In listing 7.7 the rest of the code for the datebar is shown. At line 77 a public closure iscreated to make some of the datebar’s closures accessible from outside and the datebar isalso initialized with default values.

77 this.initialize = function (afterClick , range) {78 setRangeLast(range);79 setDate($(dpFromId), db.startDate);80 setDate($(dpToId), db.endDate);81 setOnClickFunction(afterClick);82 $(actionBtnId).click(onClick);83 };84 this.setRangeLast = setRangeLast;

Listing 7.7: Private closures in datebar made public.

The problem with closures Datebar was designed as a class and because Javascriptdoesn’t have classes it was emulated via closures. However by making the variables andfunctions inside of that closure private they become inaccessible outside it hence cannotbe tested as separate units. A simple unit test like in listing 7.8 fails because it can’t callthe method which is to be tested.

1 test("dateBefore", function () {

55

7. Testing

2 var datebar = new DateBar ();3 var fixedDate = new Date (1999 , 0, 31); //31st January 19994 equals(datebar.dateBefore(fixedDate , 1),5 "Sat Jan 30 1999 00:00:00 GMT +0000 (GMT)"6 });

Listing 7.8: Simple unit test for datebar.

An argument can be made for not testing private methods directly. It can be preferableto test a class by its public methods and if there is significant functionality behind privateaccess, that might be an indicator to extrude that functionality into the public e.g. bymaking it a class [HT04]. In the case for the datebar it is logical to have the functionalitywhich is specific for it, inside its environment. Extruding it into a “class” is therefore nota suitable solution.

The problem with dates Another notable problem in unit testing the datebar wastesting code which uses dates. Fixed dates are used e.g. in new Date(2014,0,2) and thereis no problem with them in testing as they are predictable. However when using newDate() there is a problem because the time of when the call is made. This makes it difficultto test because it is not possible to know beforehand what date value is generated. Onesolution to this problem would be to extend the Date object with an option for debugmode which allows us to set the system clock manually. A call for Date.now() would thenreturn the same value independent of the time the call for the object is made. This methodcalls for an overhead in maintaining the extension although is unlikely a core functionalityof Javascript changes often. The Javascript library SinonJS offers the same functionalitywithout the hard work and in addition it works well with QUnit. In listing 7.9 is a simpleexample on how it is used. In the setup function, which is run before every test, SinonJSis used to manually set the current time to a known value (UNIX time 1). This state ispersistent through the test until the tear-down function is called after each test.

1 module("Testing DateBar", {2 /**3 * Setting up data for the unit tests4 */5 setup: function (){6 // Manually set time so Date uses our "now"7 this.clock = sinon.useFakeTimers (0, "Date");8 },9 teardown: function (){10 this.clock.restore ();11 }12 });

Listing 7.9: Example of how SinonJS is used for manually setting date

Re-factored Datebar The datebar needs to be more open for testing without pollutingthe global namespace. The DateBar closure is accessible but its private variables/closuresare not. The usage of the keyword var is the culprit and therefore it is needed to replace it.

1UNIX time is defined as the number of seconds elapsed since 1 January 1970

56


The solution is to use this instead and thereby making variables/closures accessible throughan instance of the datebar without exposing them directly to the global namespace.

The variables are assigned to the DateBar and an underscore is prepended to theirnames(see listing 7.10). That is a convention from Python to indicate that this variable isprivate and is not supposed to be used outside it is closure.

12 var DateBar = function(module_namespace){13 this._dateRangeId = "#"+ module_namespace +"DateBarDateRange";14 this._dpFromId = "#"+ module_namespace +"DateBarFrom";15 this._dpToId = "#"+ module_namespace +"DateBarTo";16 this._actionBtnId = "#"+ module_namespace +"DateBarActionButton";

Listing 7.10: Datebar refactored for testability

The closures are now prepended with this as can be seen in listing 7.11. In some cases e.g.for the onClick closure

109 this.onClick = function () {110 // The caller of this function is not datebar itself111 // therefore "this" doesn’t point to datebar.112 // Use db instead.113 db.disableAction ();114 db.onClickFunction(115 db.getDateFrom (),116 db.getDateTo (),117 db.enableAction118 );119 };120121 this._setOnClickFunction = function (onClick) {122 this.onClickFunction = onClick;123 };124125 /**126 * Initializes a datebar with pre -chosen values127 * @param afterClick128 * @param range129 */130 this.initialize = function (afterClick , range) {131 this.setRangeLast(range);132 this._setOnClickFunction(afterClick);133 $(this._actionBtnId).click(this.onClick);134 };135 };

Listing 7.11: Closures inside Datebar refactored

Now functions are accessible outside datebar and therefore it is no longer necessary toinclude them in the initialize closure in listing 7.11 to make them public. The test inlisting 7.8 now works and the various functions in datebar can be tested separately. A fullcode example for the datebar before and after being refactored is in appendix B.

57

7. Testing

7.3. System testing

System testing is performed after unit and integration testing have been completed. Thetests are structured around the GUI where CasperJS is used to emulate the user interaction.The purpose of these tests are to assert that EYK meets the user requirements and also toprevent regressions in the GUI. At this test level data is used from EYK’s database whichis used for development and contains real data that has been retrieved from customer’sdatabases. Because the database contains a mixture of data from different customers, atest database is scheduled to be created for each customer to offer a better representationof their production databases. These test databases will contain a subset of the dataavailable on the production version and have data that has either been manually pickedor created to cover possible edge cases e.g. data with missing value.

The process of creating use cases includes launching the part of EYK which is to be testedand list down how the user will interact with it. To illustrate this process the testing ofthe modules for an air quality monitor is described.

7.3.1. Test case: modules for an air quality monitor

The air quality monitor has two modules: property module and a time series module. Theformer, see fig. 7.2, is a simple module with only static data.

Figure 7.2: The property module for an air quality monitor

It shows the key properties of an asset and a picture of the asset if it is available. In thiscase no picture was available. For this module to meet its requirements it needs to:

1. Load the asset page successfully.

2. Make all the available modules for the asset accessible.

3. Fetch the correct data.

To meet item 1 a test was made which tests if the title of the asset page exists. The test

58

7.3. System testing

is made with one function from CasperJS, see listing 7.12.

9 //Load the a s s e t page10 casper . then ( func t i on ( ) {11 t e s t . a s s e r tT i t l e ( ’EYK’ , ’ a s s e t page has loaded s u c c e s s f u l l y ’ ) ;12 }) ;

Listing 7.12: System testing for air quality monitor: checking page load

The top-bar of the asset page lists all the available modules for a given asset. The airquality monitor should have two modules and therefore two tests were made to meet item 2.The tests, see listing 7.13, check for links to in the top-bar modules which belong with theasset type. The HTML elements containing these links are found using CSS selectors e.g.amodule-0 finds an <a> element that has the ID module-0. The second argument for theassert functions are strings that are used as a message when the test runs successfully.

17 // Check e lements on page18 casper . then ( func t i on ( ) {19 //Check av a i l a b l e modules in top bar20 t e s t . a s s e r tEx i s t s ( ’ a#module−0 ’ , ’ property module i s pre sent ’ ) ;21 t e s t . a s s e r tEx i s t s ( ’ a#module−1 ’ , ’ p l o t module i s pre sent ’ ) ;

Listing 7.13: System testing for air quality monitor: checking available modules

Finally to ensure that item 3 is met, the value for the ID field, displayed in the module, isevaluated (see listing 7.14). As the ID is unique and is not going to be changed, it servesas a good way to identify the asset on the page.

23 //Check i f c o r r e c t a s s e t24 var id = casper . eva luate ( func t i on ( ) { re turn $ ( ’dd : conta in s ("HVALEYRARH")

’ ) . l ength ; }) ;25 t e s t . a s se r tTrue ( id != 0 , ’The a s s e t i s c o r r e c t ’ ) ;26 }) ;

Listing 7.14: System testing for air quality monitor: checking asset ID

As these requirements are met the test scripts clicks the link to the time series module. Itis more complicated than the property module since it displays data in a dynamic way.The module is shown in fig. 7.3.

59

7. Testing

Figure 7.3: Time series module for an air quality monitor

The main purpose of the module is to provide the user a quick overview for a chosen timerange with the possibility to export raw data for further analysis. In order to meet itsrequirements the following should apply:

1. The module page loads successfully

2. The datebar controls load successfully

3. The datebar controls function properly

4. The graph is rendered with the correct data

5. The data is correct in the table

6. The data exported is correct

60

7.3. System testing

In listing 7.15 it is checked for item 1. A check is made if the title of the module existsand if it does the time series module loaded successfully.

37 //Check that the s i t e loaded s u c c e s s f u l l y38 casper . wa i tForSe l e c to r ( " div#plot_chart " , f unc t i on ( ) {39 t e s t . a s s e r tEx i s t s ( ’ d iv#p l o t_ t i t l e ’ , ’ p l o t module has loaded ’ ) ;40 }) ;

Listing 7.15: System testing for air quality monitor: check if time series module loads

For item 2 a test is made, listing 7.16, see which checks for the HTML elements of thedatebar and if they exist the test passes.

42 //Check the p l o t buttons43 casper . then ( func t i on ( ) {44 //Check i f sho r t cut buttons f o r time range are pre sent45 t e s t . a s s e r tEx i s t s ( ’ button [ on c l i c k="OAT. p l o t . dateBar . setRangeLast (30) " ] ’ ) ;46 t e s t . a s s e r tEx i s t s ( ’ button [ on c l i c k="OAT. p l o t . dateBar . setRangeLast (91) " ] ’ ) ;47 t e s t . a s s e r tEx i s t s ( ’ button [ on c l i c k="OAT. p l o t . dateBar . setRangeLast (183) " ] ’ )

;48 t e s t . a s s e r tEx i s t s ( ’ button [ on c l i c k="OAT. p l o t . dateBar . setRangeLast (365) " ] ’ )

;49 //Check i f input f o r dates are pre sent50 t e s t . a s s e r tEx i s t s ( ’ d iv#plotDateBarDateRange ’ ) ;51 t e s t . a s s e r tEx i s t s ( ’ input#plotDateBarFrom ’ ) ;52 t e s t . a s s e r tEx i s t s ( ’ input#plotDateBarTo ’ ) ;53 t e s t . a s s e r tEx i s t s ( ’ button#plotDateBarActionButton ’ ) ;54 //Check i f p l o t button i s pre sent55 t e s t . a s s e r tEx i s t s ( ’ button#plotDateBarActionButton ’ , ’ p l o t button i s

pre sent ’ ) ;

Listing 7.16: System testing for air quality monitor: check graph controls

After it has been asserted that the controls are present then their functionality can betested. In listing 7.17 the button for choosing the last month is tested. The button isclicked and the values in the input fields are evaluated to check if they have been updatedcorrectly. If the test pass then the requirements for item 3 is considered met.

58 // Cl i ck the shor t cut button59 t h i s . c l i c k ( ’ button [ on c l i c k="OAT. p l o t . dateBar . setRangeLast (30) " ] ’ ) ;60 //Check f o r changes in input f i e l d s61 var i sCorrec tDate = casper . eva luate ( func t i on ( ) {62 //Fetch va lue s from input f i e l d s63 var from = document . getElementById ( ’ plotDateBarFrom ’ ) . va lue ;64 var to = document . getElementById ( ’ plotDateBarTo ’ ) . va lue ;65 // Create expected va lue s66 var today = new Date ( ) ;67 var month_ago = new Date (new Date ( today ) . setMonth ( today . getMonth ( )−1)

) ;68 r e turn ( to == today . ddmmyyyy( ) && from == month_ago . ddmmyyyy( ) )69 }) ;70 t e s t . a s se r tTrue ( i sCorrectDate , ’ sho r t cut buttons update the date f i e l d s ’ )

;

Listing 7.17: System testing for air quality monitor: assessing graph control behaviour

Next the most important requirement of the module is tested, the graph. In listing 7.18the datebar controls are used to configure the time range and initiate the rendering of the

61

7. Testing

graph. When the graph has been rendered a screen-shot is taken of it and compared to areference image. If they match then the test passes and item 4 is met.

99 casper . wa i tWhi l eSe l ec to r ( ’#plotDateBarActionButton : d i s ab l ed ’ , f unc t i on ( ) {100 //Take p i c tu r e o f p l o t and l egends101 casper . v iewport (1024 , 768) . then ( func t i on ( ) {102 casper . c ap tu r eS e l e c t o r ( ’ air_qual ity_monitor_legend . png ’ , ’#plot_legend ’ ) ;103 casper . c ap tu r eS e l e c t o r ( ’ a ir_qual ity_monitor_plot . png ’ , ’#plot_chart>svg ’ ) ;104 }) ;105106 //Compare to r e f e r e n c e images107 casper . then ( func t i on ( ) {108 // l egends109 var l egend_re f e r ence = f s . read ( ’ . / a ir_qual i ty_monitor_legend_reference . png ’ ) ;110 var legend_new = f s . read ( ’ . / air_qual ity_monitor_legend . png ’ ) ;111 t h i s . t e s t . a s s e r t ( legend_new == legend_re fe rence , ’ a l l p l o t s e r i e s pre sent ’ ) ;112 // p l o t113 var p l o t_re f e r ence = f s . read ( ’ . / a i r_qual i ty_monitor_plot_reference . png ’ ) ;114 var plot_new = f s . read ( ’ . / air_qual ity_monitor_plot . png ’ ) ;115 t h i s . t e s t . a s s e r t ( plot_new == plot_re f e rence , ’ p l o t i s c o r r e c t ’ ) ;116 }) ;

Listing 7.18: System testing for air quality monitor: check plot output

For item 5 the first and last row in the table are checked to ensure that the data in it iscorrect. Because the time range is known beforehand, it is easy to use fixtures to assertthe values, like shown in listing 7.19.

118 //Check i f s e r i e s are pre sent in t ab l e119 casper . then ( func t i on ( ) {120 t e s t . a s s e r tEva l ( func t i on ( ) {121 r e turn $ ( "#plot_tab le tbody > t r : nth−ch i l d (1 ) > td : nth−ch i l d (2 ) : conta in s

( 0 . 1 79 ) " ) . l ength > 0 ;122 }) ;123 t e s t . a s s e r tEva l ( func t i on ( ) {124 r e turn $ ( "#plot_tab le tbody > t r : l a s t−ch i l d (1 ) > td : nth−ch i l d (1 ) : conta in s

( 1 . 8 08 ) " ) . l ength > 0 ;125 }) ;

Listing 7.19: System testing for air quality monitor: check table

The exporting of data could not be tested successfully with CasperJS even though it workedwhen manually tested on Firefox and Google Chrome. The data is exported without serverinteraction and uses the HTML5 download attribute [Moz14] which is not supported byCasperJS.

All the tests described above are contained within one Javascript file. To enable runningtests that are distributed on multiple files, shell scripts are created which are then run bythe developers before they commit their work.

7.4. Future improvements

The team at Hugfimi has set its goal on using a test first approach in its development.After researching test driven approaches the team decided to use BDD (BDD) [Nor14].

62

7.4. Future improvements

BDD is software development process where a test case is written before the code isimplemented. The developer then produces a minimum amount of code until it passes thetest. This cycle then repeats itself for each new feature. BDD differs from other test firstapproaches in that it emphasis on the behavioural aspect of the system rather then theimplementation which results in less fragile tests.

A work flow with BDD has been created at Hugfimi although the development has notyet become test driven. Hugfimi’s implementation of BDD is shown in fig. 7.4.

Figure 7.4: A flowchart for BDD.

The developer begins with creating a failing feature test and runs that test. Once it hasfailed a unit and integration test is created that also fails at first. At this point the developercan start on writing the first code. A minimum amount of code is written until it passesits test. Once the test passes then the developer evaluates if enough code has been writtento for the initial feature test. If more code is needed then a new unit and integration testis created otherwise the feature test is run with three possible outcomes:

1. Feature test fails and the focus goes back to unit and integration testing.

2. Feature test passes but more features are needed and therefore focus goes back tocreating another feature test.

63

7. Testing

3. Feature test passes and the developer evaluates that the product is complete andthe development stops.

The feature tests are written in a natural language syntax (see listing 7.20) that is providedby Behave [RJE14], a BDD test framework. The advantage of having this syntax is that itencourages developers to focus on the purpose of a feature rather than its implementation.Also the syntax makes it easier to transfer requests from customers into feature tests.

1 Feature: To be able to plot time series data23 Scenario: Time range is changed with datepickers4 Given I have loaded a plot time series module5 When I pick a date from6 And I pick a date to7 And I request a plot8 Then graph shows data for time range

Listing 7.20: Example of a use case in Behave: feature description

Listing 7.20 shows a Behave feature file and it does not contain the actual test code. Eachline between 4-8 is called a feature step and is linked to a test code through the use ofPython wrappers (e.g. @given) like shown in listing 7.21. Each step can be reused fordifferent scenarios e.g. if another scenario would use When I pick a date from then Behavewould link it to the existing test code.

1 @given("I have loaded a plot time series module")2 def step_impl(context):3 #TEST CODE HERE45 @when(’I pick a date from’)6 def step_impl(context):7 #TEST CODE HERE89 @when(’I pick a date to’)10 def step_impl(context):11 #TEST CODE HERE1213 @when(’I request a plot’)14 def step_impl(context):15 #TEST CODE HERE1617 @then(’graph shows data for time range’)18 def step_impl(context):19 #TEST CODE HERE

Listing 7.21: Example of a use case in Behave: test code

The team does not use CasperJS anymore for system tests as Behave requires the teststo be written in Python and CasperJS uses Javascript. Splinter [? ] is a testing tool thatis similar to CasperJS but uses Python instead of Javascript. It works well with Behaveand was therefore used to replace CasperJS. It also offers limited use of Javascript whichenabled most of the tests from CasperJS to be reused.

Another future improvement that Hugfimi has scheduled is having a test server automati-cally run all tests each time a developer tries to merge new code into EYK’s development

64

7.5. Summary

branch. The test server will handle the merging process for the developer i.e. accepts themerge if all test pass otherwise it rejects it. The main advantage of this setup is that testsare guaranteed to be executed. Once a developer has initialized his merge into EYK hecan therefore start working on the next task with full certainty that tests are executed.

7.5. Summary

Early in the project most of the focus was spent on implementing new features for EYKand many tasks were left waiting for testing. The emphasis however increased graduallyon testing as the project progressed and the goal is to have test driven development atHugfimi one day.

EYK is tested on three levels: unit, integration and system, and in addition Ansible handlestesting of the OS environment on the servers. EYK is tested with separate unit andintegration tests for Javascript and Python. System testing was executed with CasperJS,a Javascript library but a BDD framework is now used that uses Python and therefore asimilar tool to CasperJS was found that uses Python.

Hugfimi aims to use test driven development in the future and a test first work flow hasalready been created. Another goal is to have a test server automatically run all tests foreach change that is pushed to EYK. The test server would then accept the changes intoEYK if and only if all the tests pass.

65

8. Conclusions

8.1. Summary

This thesis presented EYK, a data management system for geographically distributeddata sources, and described its development both as a software and as a business idea.Requirements were set for the system by analyzing Gagnor, the system EYK aims toreplace. Both advantages and disadvantages of Gagnor were identified and then used asan input for EYK’s design. As EYK is designed as Gagnor’s successor, it was clear thatRE had great use for it. However to create a viable business around EYK other customerswere needed.

The Icelandic energy market was therefore researched in order to explore if other energycompanies could have use for EYK. The results from the research were that the companiesshare many requirements with RE and have plenty of use cases for EYK.

The sales model for EYK was also discussed but it has gone through two iterations. Thefirst version aimed at setting a low entry barrier by keeping the cost of one time purchaseslow. The second one was developed in cooperation with Mannvit and is simpler. In it thecustomer requests for a feature and Hugfimi responds with a cost estimate. Then if thecustomer is ready to pay for the request then it will be developed..

The development work flows were originally built on Scrum and suited the small size ofthe team for most of the time. The current work flows at Hugfimi do not share much withScrum especially after sprints were no longer used. In the beginning not much emphasiswas put on testing however as the project progressed more and more time is spent ontesting as the team tries to move to a test first approach.

The team did not have much experience in building a commercial software product andhad therefore not thought of structuring a branching model for organizing code additionsto EYK. A branching model was eventually created and has proved to be very useful.

EYK is set up at customer’s site and therefore there exists one instance for each customer.The task of setting up and maintaining these instances is likely to be time consumingunless it is automated. Therefore an emphasis was set on automating the processes forconfiguring the EYK servers. The configuration management system Ansible was used toaccomplish that task and it has enabled the team to spend less time on configuring serversand more on developing EYK.

EYK has been in constant development during the writing of this thesis and for it to cover

67

8. Conclusions

the newest state of EYK would have required frequent rewriting. This thesis is thereforeonly capable of showing a snapshot of EYK during its development. There are howeverrecent events that affect the outlook of the project and have not yet been discussed. Theseevents are the topic of next section.

8.2. Outlook

In May of 2014 Mannvit, an international engineering consulting firm, was contacted fora potential partnership. The motivation for the partnership was to increase customer’strust in the project because some of them had expressed their concerns in our ability tobe able to provide long term support for the system. Mannvit was chosen because theyare a well known consulting firm that has operations abroad where EYK has the potentialto reach a larger market. The first meeting with Mannvit took place in June and at themeeting EYK was presented. EYK got positive reception from Mannvit and it was decidedto meet again once the business plan was completed. The business plan was completed inOctober and a meeting was set to present it to Mannvit in the same month. Soon afterthat meeting, Mannvit expressed their interest for a partnership and negotiations began.

In November 2014 a consensus was reached with the clause that Mannvit joins only if acontract with RE is acquired in accordance to the business plan.

In April 2015 RE signed a contract to buy EYK and a private limited company with thename Hugfimi was founded in which both team members at Hugfimi and Mannvit arestakeholders. Hugfimi is now currently working on implementing new features for EYKthat RE has requested for.

68

References

[ABDM01] Alain Abran, Pierre Bourque, Robert Dupuis, and James W. Moore, editors.Guide to the Software Engineering Body of Knowledge - SWEBOK. IEEE Press,Piscataway, NJ, USA, 2001.

[Aga14a] Vladimir Agafonkin. Leaflet: An Open-Source Javascript Library for Mobile-Friendly Interactive Maps, 2014. URL: http://leafletjs.com/.

[Aga14b] Vladimir Agafonkin. Showcase companies using Leaflet 400, 2014. URL: https://github.com/Leaflet/Leaflet/issues/400.

[Ans14] Ansible Inc. Ansible: Let’s automate, 2014. URL: http://www.ansible.com/home.

[Aut11] National Energy Authority. Orkumál: Raforka, 2011. URL: http://www.os.is/gogn/Orkumal-arsrit/Orkumal-Raforka-2011-7-1.pdf.

[Bat14] Ned Batchelder. coverage.py, 2014. URL: http://nedbatchelder.com/code/coverage/.

[Dan15] Kevin Dangoor. TurboGears: Who’s using it?, 2015. URL: http://turbogears.org/whos-using.html.

[DP15] Massimo Di Pierro. Web2py, 2015. URL: http://www.web2py.com/.

[Dri14] Vincent Driessen. A successful Git branching model, 2014. URL: http://nvie.com/posts/a-successful-git-branching-model.

[dS14] Vanderson Mota dos Santos. Object factory for django, 2014. URL: https://github.com/vandersonmota/model_mommy.

[Eng14] Instagram Engineering. What Powers Instagram: Hun-dreds of Instances, Dozens of Technologies, 2014. URL:http://instagram-engineering.tumblr.com/post/13649370142/what-powers-instagram-hundreds-of-instances-dozens-of.

[Fou14a] Django Software Foundation. Django: Supported databases, 2014. URL: https://docs.djangoproject.com/en/dev/ref/databases/.

[Fou14b] Django Software Foundation. GeoDjango, 2014. URL: https://docs.djangoproject.com/en/dev/ref/contrib/gis/.

[Fou14c] Django Software Foundation. The MTV (or MVC) Development Pattern, 2014.URL: https://docs.djangoproject.com/en/1.6/faq/general/#.

69

http://leafletjs.com/

https://github.com/Leaflet/Leaflet/issues/400

https://github.com/Leaflet/Leaflet/issues/400

http://www.ansible.com/home

http://www.ansible.com/home

http://www.os.is/gogn/Orkumal-arsrit/Orkumal-Raforka-2011-7-1.pdf

http://www.os.is/gogn/Orkumal-arsrit/Orkumal-Raforka-2011-7-1.pdf

http://nedbatchelder.com/code/coverage/

http://nedbatchelder.com/code/coverage/

http://turbogears.org/whos-using.html

http://turbogears.org/whos-using.html

http://www.web2py.com/

http://nvie.com/posts/a-successful-git-branching-model

http://nvie.com/posts/a-successful-git-branching-model

https://github.com/vandersonmota/model_mommy

https://github.com/vandersonmota/model_mommy

http://instagram-engineering.tumblr.com/post/13649370142/what-powers-instagram-hundreds-of-instances-dozens-of

http://instagram-engineering.tumblr.com/post/13649370142/what-powers-instagram-hundreds-of-instances-dozens-of

https://docs.djangoproject.com/en/dev/ref/databases/

https://docs.djangoproject.com/en/dev/ref/databases/

https://docs.djangoproject.com/en/dev/ref/contrib/gis/

https://docs.djangoproject.com/en/dev/ref/contrib/gis/

https://docs.djangoproject.com/en/1.6/faq/general/#

REFERENCES

[Fow14] Martin Fowler. Xunit, 2014. URL: http://www.martinfowler.com/bliki/Xunit.html.

[Gag14] Gagarín. Iceland Energy Portal, 2014. URL: http://orkuvefsja.is/vefsja/orkuvefsja.html.

[Har14] Colin Harrison. Xming X Server, 2014. URL: http://www.straightrunning.com/XmingNotes/.

[HB14] Mitchell Hashimoto and John Bender. Vagrant: Developments environmentsmade easy, 2014. URL: http://www.vagrantup.com.

[Hid14] Ariya Hidayat. PhantomJS: Full web stack. No browser required, 2014. URL:http://www.phantomjs.org.

[Hip14] D. Richard Hipp. SQLite: About SQLite, 2014. URL: http://www.sqlite.org/about.html.

[HT04] Andrew Hunt and David Thomas. Pragmatic unit testing in Java with JUnit.Pragmatic Bookshelf, 2004.

[Ice14a] Iceland Construction Authority. Innri úttektir, 2014. URL: http://www.mannvirkjastofnun.is/rafmagnsoryggi/oryggisstjornun-rafveitna/rafveitur/innri-uttektir/.

[Ice14b] Iceland Geosurvey. Software: The Iceland Water Chemistry Group, 2014. URL:http://www.geothermal.is/software.

[Ice14c] Icelandic Met Office. Vatnshæðarmælakerfið, 2014. URL: http://www.vedur.is/vatnafar/vatnshaedarmaelakerfid/?ListID=0.

[Ice14d] Icelandic Met Office. Áhrif Kárahnjúkavirkjunar á vatnsborð og grunnvatn áláglendi á Héraði, 2014. URL: http://www.vedur.is/media/2012_007.pdf.

[Inc14] MongoDB Inc. MongoDB: Agile and Scalable, 2014. URL: http://www.mongodb.org/.

[Inc15] Atlassian Inc. Plan, track, work - smarter and faster, 2015. URL: https://www.atlassian.com/software/jira.

[Iwa14] IwanVosloo. Web Frameworks for Python, 2014. URL: https://wiki.python.org/moin/WebFrameworks.

[Joh14] Christian Johansen. Standalone test spies, stubs and mocks for JavaScript.No dependencies, works with any unit testing framework, 2014. URL: http://sinonjs.org/.

[JQu14] JQuery Foundation. QUnit: A Javascript Unit Testing Framework, 2014. URL:http://www.qunitjs.com.

[JW14] Lawrence Journal-World. django: The Web framework for perfectionists withdeadlines, 2014. URL: https://www.djangoproject.com/.

[Já14] Já.is. Kortavefsjá Já.is, 2014. URL: http://www.ja.is/kort/?type=map.

70

http://www.martinfowler.com/bliki/Xunit.html

http://www.martinfowler.com/bliki/Xunit.html

http://orkuvefsja.is/vefsja/orkuvefsja.html

http://orkuvefsja.is/vefsja/orkuvefsja.html

http://www.straightrunning.com/XmingNotes/

http://www.straightrunning.com/XmingNotes/

http://www.vagrantup.com

http://www.phantomjs.org

http://www.sqlite.org/about.html

http://www.sqlite.org/about.html

http://www.mannvirkjastofnun.is/rafmagnsoryggi/oryggisstjornun-rafveitna/rafveitur/innri-uttektir/



http://www.geothermal.is/software

http://www.vedur.is/vatnafar/vatnshaedarmaelakerfid/?ListID=0

http://www.vedur.is/vatnafar/vatnshaedarmaelakerfid/?ListID=0

http://www.vedur.is/media/2012_007.pdf

http://www.mongodb.org/

http://www.mongodb.org/

https://www.atlassian.com/software/jira

https://www.atlassian.com/software/jira

https://wiki.python.org/moin/WebFrameworks

https://wiki.python.org/moin/WebFrameworks

http://sinonjs.org/

http://sinonjs.org/

http://www.qunitjs.com

https://www.djangoproject.com/

http://www.ja.is/kort/?type=map

REFERENCES

[Lab14] Puppet Labs. Puppet Labs: Automate IT Ops, 2014. URL: http://www.vagrantup.com.

[Lan14a] Landsnet. Flutningskerfi Landsnets, 2014. URL: http://landsnet.is/raforkukerfid/flutningskerfilandsnets/.

[Lan14b] Landsnet. KKS kóði, 2014. URL: http://landsnet.is/raforkukerfid/flutningskerfilandsnets/kkskodi/.

[Lan14c] Landsvirkjun. Environmental Monitoring, 2014. URL: http://www.landsvirkjun.com/researchdevelopment/environmentalmonitoring/.

[Ltd14] Netcraft Ltd. Apache: Http Server Project, 2014. URL: http://httpd.apache.org/.

[Man14] Mannvit. Frummatsskýrsla 2004, 2014. URL: http://mannvit.is/media/PDF/Frummatsskyrsla_Lokaeintak_200410_a.pdf.

[Moz14] Mozilla developer network . <a>, 2014. URL: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/a.

[Nat14a] National Energy Authority. Hellisheiðarvirkjun: Eftirlit, 2014. URL:http://www.orkustofnun.is/jardhiti/skjalftar/hellisheidarvirkjun/eftirlit/.

[Nat14b] National Energy Authority. National Energy Authority Database: Boreholes,2014. URL: http://www.os.is/gogn/gogn_afhending/Borholur.

[ngi14] nginx inc. nginx, 2014. URL: http://nginx.com/.

[Nor14] Dan North. BDDWiki:BehaviourDrivenDevelopment, 2014. URL: http://behaviour-driven.org/.

[Ora14] Oracle Corporation. MySQL: The world’s most popular open source database,2014. URL: URLhttp://www.mysql.com.

[OSG14a] OSGeo. OpenLayers: Free Maps for the Web, 2014. URL: http://openlayers.org/.

[OSG14b] OSGeo. PostGIS: About PostGIS, 2014. URL: https://www.postgis.net.

[Per14] Nicolas Perriault. CasperJS is a navigating scripting & testing utility for Phan-tomJS and SlimerJS written in Javascript, 2014. URL: http://www.casperjs.org.

[Piv14] Pivotal Software Inc. RabbitMQ: Messaging that just works, 2014. URL: http://www.rabbitmq.com/.

[Pos14] PostgreSQL Global Development Group. PostgreSQL: The world’s most ad-vanced open source database, 2014. URL: http://www.postgresql.com.

[Pyt14] Python Software Foundation. unittest - Unit testing framework, 2014. URL:https://docs.python.org/2/library/unittest.html.

71



http://landsnet.is/raforkukerfid/flutningskerfilandsnets/

http://landsnet.is/raforkukerfid/flutningskerfilandsnets/

http://landsnet.is/raforkukerfid/flutningskerfilandsnets/kkskodi/

http://landsnet.is/raforkukerfid/flutningskerfilandsnets/kkskodi/

http://www.landsvirkjun.com/researchdevelopment/environmentalmonitoring/

http://www.landsvirkjun.com/researchdevelopment/environmentalmonitoring/

http://httpd.apache.org/

http://httpd.apache.org/

http://mannvit.is/media/PDF/Frummatsskyrsla_Lokaeintak_200410_a.pdf

http://mannvit.is/media/PDF/Frummatsskyrsla_Lokaeintak_200410_a.pdf

https://developer.mozilla.org/en-US/docs/Web/HTML/Element/a

https://developer.mozilla.org/en-US/docs/Web/HTML/Element/a

http://www.orkustofnun.is/jardhiti/skjalftar/hellisheidarvirkjun/eftirlit/

http://www.orkustofnun.is/jardhiti/skjalftar/hellisheidarvirkjun/eftirlit/

http://www.os.is/gogn/gogn_afhending/Borholur

http://nginx.com/

http://behaviour-driven.org/

http://behaviour-driven.org/

URL http://www.mysql.com

http://openlayers.org/

http://openlayers.org/

https://www.postgis.net

http://www.casperjs.org

http://www.casperjs.org

http://www.rabbitmq.com/

http://www.rabbitmq.com/

http://www.postgresql.com

https://docs.python.org/2/library/unittest.html

REFERENCES

[RAR14] RARIK. Ársskýrsla 2012, 2014. URL: http://www.rarik.is/skjolrarik?DocumentScreen=detail&cl=424&ccs=580.

[RJE14] Benno Rice, Richard Jones, and Jens Engel. Welcome to behave!, 2014. URL:http://pythonhosted.org/behave/.

[Rob14] Matt Robenolt. Scaling Django to 8 billion page views,2014. URL: http://blog.disqus.com/post/62187806135/scaling-django-to-8-billion-page-views.

[Sof14] Software Freedom Conservancy. –distributed-is-the-new-centralized, 2014. URL:http://git-scm.com/.

[SS14] Jeff Sutherland and Ken Schwaber. The SCRUM guide, 2014. URL: http://www.scrumguides.org/scrum-guide.html.

[Tre14] Trello Inc. Trello makes it easy to organize anything with everyone, 2014. URL:https://trello.com/home.

[Ís12] Veðurstofa Íslands. Vatnshæðarmælakerfið, 2012. URL: http://www.vedur.is/vatnafar/vatnshaedarmaelakerfid/bigimg/1460?ListID=0.

72

http://www.rarik.is/skjolrarik?DocumentScreen=detail&cl=424&ccs=580

http://www.rarik.is/skjolrarik?DocumentScreen=detail&cl=424&ccs=580

http://pythonhosted.org/behave/

http://blog.disqus.com/post/62187806135/scaling-django-to-8-billion-page-views

http://blog.disqus.com/post/62187806135/scaling-django-to-8-billion-page-views

http://git-scm.com/

http://www.scrumguides.org/scrum-guide.html

http://www.scrumguides.org/scrum-guide.html

https://trello.com/home

http://www.vedur.is/vatnafar/vatnshaedarmaelakerfid/bigimg/1460?ListID=0

http://www.vedur.is/vatnafar/vatnshaedarmaelakerfid/bigimg/1460?ListID=0

Appendices

73

A. Django

Django follows the Model-View-Controller (MVC) pattern closely enough to be calledan MVC framework but the Django foundation prefers calling it a Model-Template-View(MTV) framework [Fou14c].

(M)odel an object-relational mapper which provides a Python class based interface upona relational database, defined in models.py files.The model is a definition of thestored data in the databases,

(T)emplate a web templating system with its own templating language, used for HTML,CSS and Javascript files in static directories. The template is the definition of thepresentation of the data.

(V)iew a regular-expression-based URL dispatcher, used in urls.py. The view determineswhat data is to be displayed.

The first thing to do when starting developing in Django is to built a project. A projectis initialized by one command:

� �Django_project

manage.pyDjango_project

__init__.pysettings.pyurls.pywsgi.py

Django_app/� �Listing A.1: Django project layout

The top directory holds the name of the project which in this case is Django_project.The project management module, manage.py is also created. It’s a wrapper around thedjango-admin.py command which e.g. is used to initiate a project. The file takes care ofputting the project package on the sys.path (allows python to find projects modules) andsets the DJANGO_SETTINGS_MODULE environment variable to point to the project settingsfile.

The inner Django_project directory is the actual Python package for the project whilethe outer one is just a container.

75

A. Django

Settings and configuration for the Django project are defined in settings.py. Some ofthe topics that the settings file handles are: choice of database, path to static files fordeployment, active Django apps and more.

As noted before the URL declarations are defined in urls.py. The URL dispather usesregular expressions to match incoming request to the correct view pattern (see listing A.2.When a view is matched Django imports and calls the given view. The request is a typicalHTTPRequest object and includes HTTP headers and user-submitted data e.g. GET, POST,cookies and session data.

As an example on how requests are routed lets say we have an URL: http://eyk.hugfimi.is/asset/8 and a regular expression pattern as shown in listing A.2.

The eyk.hugfimi.is part of the URL is the root URL of the project but the part thatcomes after it is evaluated for matching. In this case /asset matches the pattern androutes the request to the view asset_default inside a view.py in a Python module calledoats.

1 from django.conf.urls import patterns , url23 urlpatterns = patterns(’oats.views’,4 url(r’^asset /(?P<asset_pk >\d+)$’, ’asset_page ’, name=’asset_default ’),

Listing A.2: Example code for URL dispatcher

In listing A.3 the matching view is shown which uses the asset_pk variable from therequest to return the default page for the requested asset.

1 def asset_page(request , asset_pk):2 asset = get_object_or_404(oats.models.Asset , pk=asset_pk)3 oat = asset.oat_instance4 return oat.get_asset_page(request , asset)

Listing A.3: Example code for view matching URL request

A.1. Apps

A Django app is required to have its own models.py file and __init__.py but other filesare optional.

A.1.1. Models

Models are Python class objects which are Object-Relational-Mapping (ORM) represen-tation of database tables. Each class defines its fields and relations. An example from

76

A.1. Apps

� �Django_app/

__init__.pymodels.pytemplates/

Django_app/html_page.html

tests.pyviews.py� �

Listing A.4: Django app layout

() for a model is shown in listing A.5. In it a class object Asset is created. It has vari-ous types of fields e.g. character based field for its name, foreign key relation to anothermodel (Assettype, Organzation) and floating integer field for its coordinates (latitude,longitude). The ORM which Django provides handles how these field are mapped to thedatabase.

1 class Asset(models.Model):2 name = models.CharField(max_length =255)3 type = models.ForeignKey(AssetType)4 sub_types = models.ManyToManyField(SubType)5 organization = models.ForeignKey(Organization)6 longitude = models.FloatField ()7 latitude = models.FloatField ()8 legacy_id = models.CharField(max_length =255, blank=True)910 def __unicode__(self):11 return self.name

Listing A.5: Example of a Django model

A.1.2. Views

The simplest description for a Django view is that it takes a web request and returns aweb response. It contains the logic for how the request is processed before a response ismade. An example of a view function was shown in listing A.3 but a better example ofthe logic a view normally contains is shown in listing A.6.

This view function handles requests for the map. The map can contain GET parametersfor coordinates, active layers on the map and the zoom level. If no parameters are inthe request then default values are used (see line 15 to 19). After the variables for theseparameters have been assigned their value they are put into a context which is a variablein a JSON format. It is sent with the request to the template which will receive theseparameters and configure the map according to their values and the HTML will then berendered.

77

A. Django

1 # Iceland fits the map2 INITIAL_LATITUDE = 653 INITIAL_LONGITUDE = -194 INITIAL_ZOOM = 75 INITIAL_BASELAYER = ’roadmap ’6 INITIAL_OVERLAYERS = ’all’789 def map_page(request):10 """11 Renders the parameters in request and returns the map html.12 :param request: a HTTPRequest13 :return: rendered html14 """15 latitude = INITIAL_LATITUDE16 longitude = INITIAL_LONGITUDE17 zoom = INITIAL_ZOOM18 base_layer = INITIAL_BASELAYER19 active_layers = INITIAL_OVERLAYERS20 try:21 latitude = float(request.GET.get[’latitude ’])22 longitude = float(request.GET.get[’longitude ’])23 zoom = int(request.GET.get[’zoom’])24 base_layer = request.GET.get(’baselayer ’)25 active_layers = request.GET.get(’layers ’)26 except TypeError:27 pass2829 context = {30 ’latitude ’: latitude ,31 ’longitude ’: longitude ,32 ’zoom’: zoom ,33 ’baselayer ’: base_layer ,34 ’layers ’: active_layers35 }36 return render(request , "map/map.html", context)

Listing A.6: Example of Django view logic

A.1.3. Templates

A template uses Django’s template language to provide a way to express presentationrather than program logic. Django requires the templates to be stored inside a direc-tory named templates. To reference a template, that path from that directory is used.For example, in listing A.4 the template html_page.html would be referenced withDjango_app/html_page.html. As can be seen having the extra directory Django_appwith the the name of module it resides in helps avoiding name conflicting when referenc-ing.

A template contains variables and tags which are used to create dynamic templates. Inlisting A.7 variables are used when calling the Javascript function initMap. They arerecognizable by their double curly brackets around them. These variables were sent to thetemplate in the context variable from the view in listing A.6. Tags are also used but theyuse the pattern {% tag %} and some require a matching end tag {% end tag %}. Theyare more complex than variables and have different functionalities for each type of tag. Inlisting A.7 the include tag is used to import another HTML file (line 8).

78

A.1. Apps

1 <html lang="en">2 <head >3 <title >EYK </title >4 <meta charset="UTF -8">5 <meta name="viewport" content="width=device -width , initial -scale =1.0">6 </head >7 <body >8 {% include "eyk/header.html"%}9 <div id="map" ></div >10 <script src="{% static "map/js/map.js" %}"></script >11 <script type=’text/javascript ’>12 initMap ({{ latitude }}, {{ longitude }}, {{ zoom }}, ’{{ baselayer }}’);13 $(document).ready(function (){14 addMarkers(’{{ layers }}’);15 });16 </script >17 </body >18 </html >

Listing A.7: Example of a Django template

79

B. Datebar refactored: Full code

1 Date.prototype.ddmmyyyy = function () {2 var yyyy = this.getFullYear ().toString ();3 var mm = (this.getMonth ()+1).toString (); // getMonth () is zero -based4 var dd = this.getDate ().toString ();5 return (dd[1]?dd:"0"+dd[0]) +’-’+ (mm[1]?mm:"0"+mm[0]) +’-’+ yyyy;6 };78 var DateBar = function(module_namespace){9 var dateRangeId = "#"+ module_namespace +"DateBarDateRange";10 var dpFromId = "#"+ module_namespace +"DateBarFrom";11 var dpToId = "#"+ module_namespace +"DateBarTo";12 var actionBtnId = "#"+ module_namespace +"DateBarActionButton";13 $(dateRangeId).datepicker ({14 format: ’dd-mm -yyyy’,15 todayBtn: "linked",16 language: "is",17 calendarWeeks: true ,18 autoclose: true ,19 todayHighlight: true20 });2122 var getDate = function($picker){23 return $picker.datepicker(’getDate ’).valueOf ();24 };2526 var db = this;27 var getDateFrom = function (){28 return getDate($(dpFromId));29 };3031 var getDateTo = function (){32 return getDate($(dpToId));33 };3435 var setDate = function($picker , date){36 $picker.datepicker(’setDate ’, date);37 $picker.datepicker(’update ’);38 };3940 var dateBefore = function(to , days){41 return new Date(42 to.getFullYear (),43 to.getMonth (),44 to.getDate () - days);45 };4647 var disableAction = function (){48 $(actionBtnId).prop(’disabled ’, true);49 };5051 var enableAction = function (){52 $(actionBtnId).prop(’disabled ’, false);53 };54 var setRangeLast = function(days) {55 db.endDate = new Date();56 db.startDate = dateBefore(db.endDate , days);57 setDate($(dpFromId), db.startDate);58 setDate($(dpToId), db.endDate);59 };

81


60 var onClick = function () {61 disableAction ();62 db.onClickFunction(63 getDateFrom (),64 getDateTo (),65 enableAction66 );67 };68 var setOnClickFunction = function(onClick) {69 db.onClickFunction = onClick;70 };7172 // Make selected closures public73 this.initialize = function (afterClick , range) {74 setRangeLast(range);75 setDate($(dpFromId), db.startDate);76 setDate($(dpToId), db.endDate);77 setOnClickFunction(afterClick);78 $(actionBtnId).click(onClick);79 };80 this.setRangeLast = setRangeLast;81 this.onClick = onClick;82 this.setOnClickFunction = setOnClickFunction;83 };

Listing B.1: Datebar before refactoring

1 Date.prototype.ddmmyyyy = function () {2 var yyyy = this.getFullYear ().toString ();3 var mm = (this.getMonth ()+1).toString (); // getMonth () is zero -based4 var dd = this.getDate ().toString ();5 return (dd[1]?dd:"0"+dd[0]) +’-’+ (mm[1]?mm:"0"+mm[0]) +’-’+ yyyy;6 };78 var DateBar = function(module_namespace){9 this._dateRangeId = "#"+ module_namespace +"DateBarDateRange";10 this._dpFromId = "#"+ module_namespace +"DateBarFrom";11 this._dpToId = "#"+ module_namespace +"DateBarTo";12 this._actionBtnId = "#"+ module_namespace +"DateBarActionButton";1314 var db = this; // Hold reference to this --> datebar1516 $(this._dateRangeId).datepicker ({17 format: ’dd-mm -yyyy’,18 todayBtn: "linked",19 language: "is",20 calendarWeeks: true ,21 autoclose: true ,22 todayHighlight: true23 });2425 /**26 * Gets the date from the HTML element which corresponds to $picker27 * @param $picker28 * @returns {XMLList|XML|Namespace |*| Array|boolean}29 */30 this.getDate = function ($picker) {31 return $picker.datepicker(’getDate ’).valueOf ();32 };3334 /**35 * Gets the start date from the template36 * @returns {XMLList|XML|Namespace |*| Array|boolean}37 */38 this.getDateFrom = function () {39 return this.getDate($(this._dpFromId));40 };4142 /**43 * Gets the end date from the template44 * @returns {XMLList|XML|Namespace |*| Array|boolean}

82

45 */46 this.getDateTo = function () {47 return this.getDate($(this._dpToId));48 };4950 /**51 * Sets/Updates the date for an HTML element52 * @param $picker The HTML element53 * @param {object} Date The date which the HTML element will be updated to54 */55 this.setDate = function ($picker , date) {56 $picker.datepicker(’setDate ’, date);57 $picker.datepicker(’update ’);58 };596061 /**62 * Disables the button which draws the plot63 */64 this.disableAction = function () {65 $(db._actionBtnId).prop(’disabled ’, true);66 };6768 /**69 * Enables the button which draws the plot70 */71 this.enableAction = function () {72 $(db._actionBtnId).prop(’disabled ’, false);73 };7475 /**76 * Sets/Updates the start and end date. End date is today and77 * @param {int} days Number of days the date range spans78 */79 this.setRangeLast = function (days) {80 this.setEndDate(new Date());8182 var startDate = new Date(this.endDate);83 startDate.setDate(this.endDate.getDate ()-days);84 this.setStartDate(startDate);85 };8687 /**88 * Updates the end date of datebar89 * @param date90 */91 this.setEndDate = function (date) {92 this.endDate = date;93 this.setDate($(this._dpToId), this.endDate);94 }9596 /**97 * Updates the starting date of datebar98 * @param {object} date99 */

100 this.setStartDate = function (date) {101 this.startDate = date;102 this.setDate($(this._dpFromId), this.startDate);103 }104105 this.onClick = function () {106 // The caller of this function is not datebar itself107 // therefore "this" doesn’t point to datebar.108 // Use db instead.109 db.disableAction ();110 db.onClickFunction(111 db.getDateFrom (),112 db.getDateTo (),113 db.enableAction114 );115 };116117 this._setOnClickFunction = function (onClick) {

83


118 this.onClickFunction = onClick;119 };120121 /**122 * Initializes a datebar with pre -chosen values123 * @param afterClick124 * @param range125 */126 this.initialize = function (afterClick , range) {127 this.setRangeLast(range);128 this._setOnClickFunction(afterClick);129 $(this._actionBtnId).click(this.onClick);130 };131 };

Listing B.2: Datebar after refactoring

84

C. Ansible

Ansible requirements are only those that the system running it has Python and SSHinstalled, both which are pre-installed on most modern OSs. A requirement for Hugfimito be able to monitor and maintain EYK quickly and efficiently is to have a Secure-Shell(SSH) connection to its servers either directly or through a VPN connection. A simplediagram of a possible setup at RE is shown in fig. C.1. The connection enables developersto maintain the servers from anywhere.

Figure C.1: How Hugfimi connects to customer’s instances of EYK

An example of a Ansible project layout is shown in listing C.1. To run this project, onesimple command is needed:

ansible-playbook -i hosts site.yml

The -i flag is to indicate the inventory file, Hosts, and site.yml is the main playbook. Bothof these concepts and many more are explained in the following text.

Hosts The hosts file contains the addresses to the systems which Ansible maintains. Thesystems can be grouped together so common configurations can be set for them. A demon-stration of this grouping is shown in listing C.2 where the address (109.222.223.153) isgrouped in webserver.

85

C. Ansible

� �. # top level of an ansible instance|-- webserver.yml # main playbook for the webserver group|-- hosts # hosts and groups of hosts~(ip addresses , user names etc.)|-- roles # a level of abstraction for organizing playbooks

|-- eyk|-- files # files that need to be transferred to or run on the hosts|-- handlers # tasks that are run on events|-- meta # contains files that establish role dependencies|-- tasks # all tasks a playbook runs.|-- templates # files that use variables to substitute information during

creation|-- vars # variables for the roles , can be used in configuration files

|-- site.yml� �Listing C.1: Ansible project layout

1 [ webserver ]2 109 . 222 . 223 . 153 ipaddre s s =109.222 .223 .153

Listing C.2: The inventory file (hosts) for Ansible project

It is also possible to bind variables to a single host like is done with ip address but theycan also be bound to groups e.g. by adding the lines in listing C.3 to Hosts would make thevariable operating_system and its value accessible in all tasks which are run for webserver.

1 [ webserver :vars ]2 operating_system="CentOS 6.4"

Listing C.3: Variables inside an inventory file

Playbooks A playbook is where the design of the configuration management is set. Itcontains the mapping of tasks and roles to hosts or groups and the order in which theseactions are performed. The main playbook is site.yml. In it the entire infrastructure ofthe project is defined. This is done by assigning roles and tasks to the hosts and groupsdefined in the inventory file (Hosts). To organize further, each group will have its ownplaybook which site.yml will include as can be seen in listing C.4 where webserver.yml isincluded.

5 ---6 # Main playbook7 - i n c lude : webserver . yml

Listing C.4: The main playbook

In listing C.5 is the playbook for the webserver group. The value for hosts defines thegroup which the playbook is run on. The user is the user on the host machine which is usedto run the tasks and finally the roles are a collection of playbooks with specific purposeswhich are separated to offer more reuse.

86

1 ---2 # Playbook for webservers3 − host s : webserver4 user : vagrant5 r o l e s :6 - apache7 - po s tg r e s8 - django

Listing C.5: The main playbook for EYK demo

Roles Roles are used to group content. A directory layout for a role is shown in listing C.1where each directory has a special function of a role. In the tasks directory there areplaybooks which are just collection of tasks run by the same user. The role handles thereferences from these tasks for the files in the files directory. If a role has a dependencyon another role then that relation can be declared inside the meta directory. The varsdirectory holds the variables which are used in that role. If there is a conflict of variablenames then the role variable has precedence over variables defined in hosts or in playbooksat a higher abstract level. The full order of variable precedence is:

1. Variables declared at the command line using -e flag

2. Variables defined in playbooks. The variable closest in scope to the task has prece-dence e.g. a role variable has precedence over a variable defined in the main play-book (site.yml).

3. Variables defined in inventory file (hosts).

4. Facts discovered about the system. These are variables derived from the OS.

5. “Role default variables” which are used to set values to included or dependent roles.

The handlers directory includes playbooks but differs from the playbooks in the tasksdirectory by being event driven. The handler is triggered when a regular task is run andhas a notify directive, e.g. if a configuration file is changed by a regular task then the tasknotifies the handler of the change which restarts the service which uses the configurationfile.

The templates directory contains files which are transfered to the remote system. Whatseparates templates from files in the files directory is that they use a template engine,similar to Django (see appendix A), for logical constructs (loops, if-statements) and sub-stitute variables. The same template can therefore be used for different hosts in which forfiles in files directory would need a separate file for each occasion.

87

D. Deployment to demo server: Fullexample

Assume, the server has the ip-address 188.226.254.155 and in Ansible it is grouped underdemo (see listing D.1).

1 [ demo ]2 188 . 226 . 254 . 155 ipaddre s s =188.226 .254 .155

Listing D.1: Ansible host file

Playbooks make it easier to reuse by separating host specific configurations into an in-ventory file. The inventory for the demo is shown in listing D.2. In it the rootuser isdefined because in some cases the user root is not available. The database interfaces, baseextensions and modules which are enabled for the server are defined from line 5 to 15.From line 17 to 21 the configurations are defined for EYK’s database.

1 ---2 # Variables specific for DEMO3 r oo tu s e r : root45 db_inte r face s :6 - db_inte r face s . db_or . bo r eho l e s7 - db_inte r face s . db_or . a s s e tda ta8 - db_inte r face s . db_or . a i r q u a l i t y910 #base_extensions:1112 modules:13 - modules . u t i l s14 - modules . p l o t15 - modules . p r op e r t i e s1617 database:18 name: y ggd r a s i l19 user : apo l l o20 al lowed_hosts :21 - hugf imi . com

Listing D.2: The group variables for the demo server

The main playbook for the demo is shown in listing D.3. In it are roles assigned to theserver.

89

D. Deployment to demo server: Full example

1 ---2 # Playbooks for EYK demos3 − host s : demo4 user : {{ roo tu s e r }}5 r o l e s :6 - common78 − host s : demo9 user : vagrant10 r o l e s :11 - apache12 - po s tg r e s13 - django14 - demo

Listing D.3: The roles used for configuring the demo server

90

D.1. Role: common

D.1. Role: common

When the machine is initialized it is only accessible with a SSH connection using the rootuser credentials.

1 ---2 # Tasks to setup main user3 − name: System | User | c r e a t e user groups4 group: name={{ item . key }} s t a t e={{ item . value . s t a t e }}5 with_dict: user_groups6 tags :7 - product ion8 - development910 − name: System | User | c r e a t e u s e r s11 user : name={{ item . key }}12 groups={{ item . value . groups }}13 s t a t e={{ item . value . s t a t e }}14 with_dict: u s e r s15 tags :16 - product ion17 - development1819 − name: System | SSH | c r e a t e . ssh d i r e c t o r y f o r user vagrant20 f i l e : des t=/home/vagrant / . ssh21 owner=vagrant22 group=vagrant23 s t a t e=d i r e c t o r y24 tags :25 - s sh26 - product ion27 - development2829 − name: System | SSH | add deve loper ssh keys to authorized_keys30 authorized_key: user=vagrant31 key="{{ item }}"32 wi th_f i l e :33 - ssh_keys34 tags :35 - s sh36 - product ion37 - development

Listing D.4: Ansible tasks for the common role

The common role is is run by root because that is the only user in system when it isexecuted. However after the tasks have executed successfully the user vagrant has beencreated with SSH access along the user groups shown in listing D.4.

91


D.2. Role: apache

1 ---2 # Tasks for setting up Apache3 − name: Apache | i n s t a l l Apache and wsgi i n t e r f a c e4 remote_user: "{{ rootuser }}"5 sudo: yes6 yum: name={{ item }} s t a t e=present7 with_items:8 - httpd9 - mod_wsgi10 tags :11 - product ion

Listing D.5: Ansible tasks for the apache role: install Apache

Tasks for the webserver are executed in the apache role. First tasks, see listing D.5, involveinstalling Apache and creating the directory in which static files are stored.

24 − name: Apache | add apache user to deve loper group25 remote_user: "{{ rootuser }}"26 sudo: yes27 user : append=yes28 name=apache29 groups=deve loper30 tags :31 - product ion

Listing D.6: Ansible tasks for the apache role: Configuring user apache

The user apache is then added (listing D.6) to the developer group to grant him the sameaccess rights as the vagrant user.

92

D.2. Role: apache

33 − name: Apache | s e t c on f i g u r a t i o n s f o r s i t e s34 remote_user: "{{ rootuser }}"35 sudo: yes36 copy: s r c=r o l e s /apache/ f i l e s /{{ item }}37 dest=/etc /httpd /{{ item }}38 owner=root39 group=root40 mode=64441 with_items:42 - conf /43 - conf . d/44 - s i t e s −av a i l a b l e /45 - s i t e s −enabled /46 tags :47 - product ion48 no t i f y :49 - r e s t a r t apache5051 − name: Apache | symlink s i t e s −a v a i l a b l e −> s i t e s −enabled52 remote_user: "{{ rootuser }}"53 sudo: yes54 f i l e : s r c=/etc /httpd/ s i t e s −av a i l a b l e /{{ item }}55 dest=/etc /httpd/ s i t e s −enabled /{{ item }}56 owner=root57 group=root58 mode=064459 s t a t e=l i n k60 with_items: s i t e s_enab l ed61 tags :62 - product ion63 no t i f y :64 - r e s t a r t apache

Listing D.7: Ansible tasks for the apache role: Settings up sites

The final two tasks (listing D.7) configure the webserver, first by copying the configurationfiles from the repository to the server and then by activating the enabled configurations,shown in listing D.8, by linking EYK’s configurations to the directory sites_enabled.

1 ---2 # Variables set for the production role34 s i t e s_enab l ed :5 - eyk . conf

Listing D.8: Ansible variables for the apache role

In cases where it is not the initial run of the webserver a restart of the server needs to bemade in order for changes to register. A handler, listing D.9, is therefore created whichexecutes when changes are made.

93


1 ---2 − name: r e s t a r t apache3 remote_user: "{{ rootuser }}"4 sudo: yes5 s e r v i c e : name=httpd s t a t e=r e s t a r t e d67 − name: r e s t a r t i p t a b l e s8 ac t i on : s e r v i c e name=i p t a b l e s s t a t e=r e s t a r t e d

Listing D.9: Ansible handlers for the Apache role

94

D.3. Role: postgres

D.3. Role: postgres

The tasks for setting up PostgreSQL are in the postgres role. Not included are specialconfigurations for EYK which are under another role, django.

1 ---2 − name: PostgreSQL | Yum | exc lude o ld ve r s i on3 remote_user: "{{ rootuser }}"4 sudo: yes5 copy: s r c=r o l e s / po s tg r e s / f i l e s /CentOS−Base . repo6 dest=/etc /yum. repos . d/CentOS−Base . repo7 owner=root8 group=root9 tags :10 - database11 - product ion12 - development1314 − name: PostgreSQL | Yum | add r epo s i t o r y f o r new ve r s i on15 remote_user: "{{ rootuser }}"16 sudo: yes17 yum: name=’http :// yum.postgresql.org /9.2/ redhat/rhel -6-x86_64/pgdg -centos92 -9.2 -6.

noarch.rpm’18 s t a t e=pre sent19 tags :20 - database21 - product ion22 - development2324 − name: PostgreSQL | Yum | i n s t a l l PostgreSQL25 remote_user: "{{ rootuser }}"26 sudo: yes27 yum: name={{ item }} s t a t e=present28 with_items:29 - po s t g r e sq l 9230 - po s tg re sq l92−con t r i b31 - po s tg re sq l92−deve l32 - po s tg re sq l92− l i b s33 - po s tg re sq l92−s e r v e r34 tags :35 - database36 - product ion37 - development

Listing D.10: Ansible tasks for the postgres role

CentOS contains postgres packages within its default repositories but to ensure that thelatest version (at that time) of it is available the packages from the postgres project itselfare used. In listing D.10 postgres is excluded from YUM’s default repo and the postgresrepo is added. This ensures that extensions and other packages related to postgres are notmismatched because of a difference in packages between the default repo and the postgresrepo.

95


50 − name: PostgreSQL | add pos tg r e s user to deve loper group51 remote_user: "{{ rootuser }}"52 sudo: yes53 user : append=yes54 name=pos tg r e s55 groups=deve loper56 tags :57 - database58 - product ion59 - development

Listing D.11: Ansible tasks for the postgres role: Setting up user postgres

The postgres user is set up in listing D.11. That user is granted necessary access privilegesto the file system to run all intended tasks.

62 − name: PostgreSQL | I n i t i a l i z e database c l u s t e r63 remote_user: "{{ rootuser }}"64 sudo: yes65 s h e l l : runuser − l po s t g r e s −c ’/usr/pgsql -9.2/ bin/initdb --locale=en_US.UTF -8 --

encoding=UTF -8’66 c r e a t e s=/var / l i b / pgsq l /9 .2/ data/pg_hba . conf67 tags :68 - database69 - product ion70 - development

Listing D.12: Ansible tasks for the postgres role: Initialize database cluster

After that is done the database cluster is initialized (see listing D.12) and in it the databasefor EYK will be created.

72 − name: PostgreSQL | a l low l o c a l a c c e s s to database73 remote_user: "{{ rootuser }}"74 sudo: yes75 copy: s r c=r o l e s / po s tg r e s / f i l e s /pg_hba . conf76 dest=/var / l i b / pgsq l /9 .2/ data/pg_hba . conf77 owner=pos tg r e s78 group=pos tg r e s79 backup=yes80 no t i f y : s t a r t PostgreSQL81 tags :82 - database83 - product ion84 - development

Listing D.13: Ansible tasks for the postgres role

In listing D.13 the last task copies the configuration file for EYK from the Ansible repositoryto the server. After that is done the server can be started by notifying its handler.

96

D.4. Role: django

D.4. Role: django

1 # Django tasks2 ---3 - i n c lude : g i t . yml4 - i n c lude : yum. yml5 - i n c lude : pip . yml6 - i n c lude : database . yml7 - i n c lude : django . yml8 - i n c lude : common . yml

Listing D.14: List of subroles for the Django role

At this point the webserver and database have been installed and configured for theenvironment. Now EYK’s repository can be fetched and configured. The role django isintended for tasks which act on EYK’s code. There are many tasks involved and thereforethey are split into multiple files like shown in listing D.14. These “subroles” are describedbelow in more detail.

1 ---2 − name: System | Permiss ions | c r e a t e / vagrant / s r c d i r e c t o r y3 remote_user: "{{ rootuser }}"4 sudo: yes5 f i l e : des t=/vagrant / s r c6 owner=vagrant7 group=vagrant8 s t a t e=d i r e c t o r y9 tags :10 - product ion11 - development12 - c l one1314 − name: Django | Git | c l one EYK repo s i t o r y15 sudo: no16 g i t : repo=git@github . com:hgy l f a son /eyk . g i t17 dest=/vagrant / s r c18 accept_hostkey=yes19 update=no20 ve r s i on=master21 tags :22 - product ion23 - EYK24 - c l one25 no t i f y :26 - Django | S e t t i n g s | update s e t t i n g s f i l e

Listing D.15: Ansible tasks for the git part in the django role

Before any actions can be made on the code it first needs to be pulled from the server.First in listing D.15 the directory for EYK’s repository is created and then the head ofthe master branch is pulled.

97


1 ---2 − name: Django | S e t t i n g s | update s e t t i n g s f i l e3 remote_user: "{{ rootuser }}"4 sudo: yes5 template : s r c=r o l e s /django/ f i l e s / product ion . py . j 26 dest=/vagrant / s r c /eyk/eyk/ s e t t i n g s / product ion . py7 owner=vagrant8 group=use r s

Listing D.16: Ansible handler for the git part in the Django role

After the pull, a handler (listing D.16) is activated which renders the settings template (list-ing D.17) with the variables shown in listing D.2. For deployment to customer’s serverthis is used to activate the features which they have purchased.

1 # coding: utf -823 from . base import ∗45 DEBUG = False6 TEMPLATE_DEBUG = DEBUG78 ALLOWED_HOSTS = [9 {% f o r host in database . a l lowed_hosts %}10 u’{{ host }}’ ,11 {% endfor %}12 ]1314 DATABASES = {15 ’default ’: {16 ’ENGINE ’: ’django.db.backends.postgresql_psycopg2 ’ ,17 ’NAME’: ’{{ database.name }}’ ,18 ’USER’: ’{{ database.user }}’ ,19 ’PASSWORD ’: ’{{ database.password }}’ ,20 ’HOST’: ’{{ database.host | default(’’) }}’ ,21 ’PORT’: ’{{ database.port | default(’’) }}’ ,22 }23 }242526 INSTALLED APPS += (27 # installed database interfaces28 {% f o r i n t e r f a c e in db_inte r face s %}29 ’{{ interface }}’ ,30 {% endfor %}3132 # installed base extensions33 {% f o r ex tens i on in base_extens ions | d e f au l t ( ’’ ) %}34 ’{{ extension }}’ ,35 {% endfor %}3637 # installed modules38 {% f o r module in modules %}39 ’{{ module }}’ ,40 {% endfor %}41 )

Listing D.17: Ansible template for Django settings in production environment

EYK requires some packages (see listing D.18) which are only available with YUM. Thosethat are only used in development environment are put under another task which is onlyused for development.

98

D.4. Role: django

1 − name: Django | Yum | i n s t a l l packages ( a l l )2 remote_user: "{{ rootuser }}"3 sudo: yes4 yum: name={{ item }} s t a t e=present5 with_items:6 - python−psycopg27 - python−pycur l8 - python−deve l9 - subver s i on10 - gcc−c++11 - s q l i t e −deve l12 - l ibxml2−deve l13 - swig14 - expat−deve l15 - l i b c u r l −deve l16 - python−matp lo t l i b17 - gdal−python18 - rabbitmq−s e r v e r19 - pymssql2021 − name: Django | Yum | i n s t a l l packages ( development )22 remote_user: "{{ rootuser }}"23 sudo: yes24 yum: name={{ item }} s t a t e=present25 with_items:26 - ’graphviz*’27 tags :28 - development

Listing D.18: Ansible tasks for the YUM part in the Django role

99


1 ---2 − name: System | get EPEL r epo s i t o r y f o r Centos ( nece s sa ry f o r pip )3 remote_user: "{{ rootuser }}"4 sudo: yes5 yum: name=http://www. f edora . i s / f edora / epe l /6/x86_64/ epel−r e l e a s e −6−8.noarch . rpm6 s t a t e=pre sent7 tags :8 - product ion9 - development1011 − name: System | i n s t a l l pip the package manager f o r python12 remote_user: "{{ rootuser }}"13 sudo: yes14 yum: name=python−pip15 s t a t e=pre sent16 tags :17 - product ion18 - development1920 − name: Django | Pip | i n s t a l l packages ( base )21 remote_user: "{{ rootuser }}"22 sudo: yes23 pip: requ i rements={{ django_app_path }}/ requi rements /base . txt24 s t a t e=pre sent25 tags :26 - development27 - product ion2829 − name: Django | Pip | i n s t a l l packages ( product ion )30 remote_user: "{{ rootuser }}"31 sudo: yes32 pip: requ i rements={{ django_app_path }}/ requi rements / product ion . txt33 s t a t e=pre sent34 tags :35 - product ion3637 − name: Django | Pip | i n s t a l l packages ( development )38 remote_user: "{{ rootuser }}"39 sudo: yes40 pip: requ i rements={{ django_app_path }}/ requi rements / l o c a l . txt41 s t a t e=pre sent42 tags :43 - development

Listing D.19: Ansible tasks for the pip part in the Django role

100

D.4. Role: django

In the pip role, see listing D.19, the package manager is installed and the python packagesfor EYK are installed using requirements file from the EYK repository. There are threetypes of requirement files: base, local and production. The base file contains packageswhich are used in development and production environment, local is only for developmentand production likewise for production.

1 ---2 − name: PostgreSQL | Star t s e r v e r3 remote_user: "{{ rootuser }}"4 sudo: yes5 s e r v i c e :6 name=pos tg r e sq l −9.27 s t a t e=s t a r t ed8 enabled=yes910 − name: PostgreSQL | c r e a t e user apo l l o11 remote_user: "{{ rootuser }}"12 sudo: yes13 pos tg re sq l_use r : name=apo l l o14 l og in_user=pos tg r e s15 r o l e_at t r_ f l ag s=CREATEDB,CREATEROLE16 s t a t e=pre sent17 tags :18 - database19 - product ion20 - development21 - apo l l o2223 − name: PostgreSQL | I n i t i a l i z e y g gd r a s i l database24 remote_user: "{{ rootuser }}"25 sudo: yes26 sudo_user: po s t g r e s27 postgresql_db: name=yggd r a s i l28 owner=apo l l o29 port=543230 s t a t e=pre sent31 encoding=’UTF -8’32 template=’template0 ’33 tags :34 - database35 - product ion36 - development3738 − name: PostgreSQL | add hs to r e extens i on to yggd r a s i l database39 remote_user: "{{ rootuser }}"40 sudo: yes41 sudo_user: po s t g r e s42 s h e l l : / usr / pgsql −9.2/ bin / psq l −U pos tg r e s −d yggd r a s i l −c ’CREATE EXTENSION IF NOT

EXISTS hstore;’43 tags :44 - database45 - product ion46 - development

Listing D.20: Ansible tasks for the database part in the Django role

In the django role are also set of tasks (see listing D.20) for configuring the database forEYK. The first task is repetition from the postgres role but as Ansible does not executetasks unless needed the task serves as a guarantee that the database is running. Thefollowing tasks require that the database is up as they create a new user (with certainprivileges) and initialize the EYK database with the necessary extensions.1

1In the postgres role the database cluster was initialized but not the database for EYK.

101


1 ---2 − name: Django | manage . py | c o l l e c t s t a t i c3 sudo: no4 django_manage: command=c o l l e c t s t a t i c5 app_path={{ django_app_path }}6 tags :7 - django8 - prepare9 no t i f y :10 - r e s t a r t Apache1112 − name: Django | manage . py | syncdb13 sudo: no14 django_manage: command=syncdb15 app_path={{ django_app_path }}16 tags :17 - django18 - prepare

Listing D.21: Ansible tasks for Django part in the django role

Django automatically handles the static files while in debug mode but not in productionmode. When switching to production the static files need to be collected into a directorywhich Apache has been configured to use to serve static files. The collectstatic task inlisting D.21 serves this purpose and ensures also that changes are registered right awayby notifying a handler to restart Apache. The latter task builds the database from theDjango models belonging to the enabled apps. The only thing left for the database now isto populate it with appropriate data but that is executed in the demo role.

The last task for the django role sets up the logging (see listing D.22). A directory andan empty log file are created with the correct permissions so Django can log into the file.

1 ---2 − name: Django | Logging | c r e a t e l og d i r e c t o r y and f i l e3 remote_user: "{{ rootuser }}"4 sudo: yes5 f i l e : path=/var / log /{{ item . name }}6 group=deve loper7 owner=vagrant8 mode=07709 s t a t e={{ item . s t a t e }}10 with_items:11 - { name: django , s t a t e : d i r e c t o r y }12 - { name: django/eyk . hugf imi . i s . log , s t a t e : touch }13 tags :14 - product ion15 - l og

Listing D.22: Ansible tasks for the common part in the django role

102

D.5. Role: demo

D.5. Role: demo

1 ---2 − name: Demo | populate database | copy dump ( part I )3 remote_user: "{{ rootuser }}"4 sudo: yes5 copy: s r c=r o l e s /demo/ f i l e s /dump/6 dest=/tmp/dump/7 owner=root8 group=root9 tags :10 - demo11 - database1213 − name: Demo | populate database | combine data f i l e s ( part I I )14 remote_user: "{{ rootuser }}"15 sudo: yes16 s h e l l : cat y g gd r a s i l ∗ > y g g d r a s i l a l l17 chd i r=/tmp/dump/18 c r e a t e s=/tmp/dump/ y g g d r a s i l a l l19 tags :20 - demo21 - database2223 − name: Demo | populate database | populate ( part I I I )24 remote_user: "{{ rootuser }}"25 sudo: yes26 s h e l l : p sq l −U pos tg r e s −d yggd r a s i l −f /tmp/dump/ y g g d r a s i l a l l27 no t i f y :28 - c leanup database dump29 tags :30 - demo31 - database3233 − name: Rabbitmq | s t a r t s e r v e r34 remote_user: "{{ rootuser }}"35 sudo: yes36 s e r v i c e :37 name=rabbitmq−s e r v e r38 s t a t e=s t a r t ed39 enabled=yes40 tags :41 - product ion

Listing D.23: Ansible tasks for the demo role

The last role is specific for the demo server hence the name demo. The tasks in listing D.23are three steps in populating the database. A data dump is stored in multiple partsin the repository and are copied to the server. There they are combined into one filewhich is executed on the database resulting in populating it. The last task starts theRabbitMQ [Piv14] server which it handles queuing tasks for scraping data from customersdatabases into EYK’s database.

The final result, shown in fig. D.1, is a demo running the latest version of EYK with adatabase populated with official data.

103


Figure D.1: The start page for the EYK demo.

104

development of eyk - skemman · facultyofindustrialeng,mechanicaleng.andcomputerscience...

Documents