A PROOF OF CONCEPT FOR PEOPLETRACKING APPLICATION USING

SMART-SPACES, ONLING SERVICESAND COMPUTER VISION

Frank Wickström

Master of Science ThesisSupervisor: Prof. Johan LiliusAdvisor: Dr. Sébastien Lafond

Department of Information TechnologiesÅbo Akademi University

March 2013

ABSTRACT

People share more information about themselves and other than ever before and mostof the data stored by services are not used to the extent that they could be. This hasled to an increased popularity in data mining and knowledge extraction, where existingdata is being used to create intelligent systems and artificial intelligence. Smart-spaces,where data is used to create an environment where devices and humans interact are alsoon the rise as the internet of things becomes more popular and devices are becomingaware of their environments.

In this thesis we present an implementation of how to track people given existingsources of data. We cover how we combine data sources, and make them aware of eachother by connecting them to a single database. The created software is able to trackpeople based on information from web services as well as IP cameras, and is managedvia a web interface. We have come to the conclusion that the information alreadyavailable today could be used to create new and smart applications if combined andprocessed the right way.

Keywords: people tracking, smart-spaces, data mining, smart-m3

i

CONTENTS

Abstract i

Contents ii

List of Figures iv

1 Introduction 11.1 AMEBA project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Thesis structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 People tracking 42.1 Tracking techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.1.1 Human recognition . . . . . . . . . . . . . . . . . . . . . . . 52.1.2 Electronic tracking . . . . . . . . . . . . . . . . . . . . . . . 62.1.3 Location prediction and data mining . . . . . . . . . . . . . . 72.1.4 Web services . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.2 Ethics, intrusiveness and ignorance . . . . . . . . . . . . . . . . . . . 8

3 Choosing and combining technologies 103.1 Application platform . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.1.1 Web framework . . . . . . . . . . . . . . . . . . . . . . . . . 113.1.2 Web application and background tasks . . . . . . . . . . . . . 113.1.3 Web server . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.2 Online APIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.3 Databases and media storage . . . . . . . . . . . . . . . . . . . . . . 14

3.3.1 Smart-M3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143.3.2 SQL and NoSQL combination . . . . . . . . . . . . . . . . . 153.3.3 Image and video storage . . . . . . . . . . . . . . . . . . . . 16

3.4 Face detection and recognition . . . . . . . . . . . . . . . . . . . . . 16

4 Implementation 184.1 Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184.2 Krang Demonstrator . . . . . . . . . . . . . . . . . . . . . . . . . . 18

ii

4.2.1 Functionality . . . . . . . . . . . . . . . . . . . . . . . . . . 194.3 Implementation overview . . . . . . . . . . . . . . . . . . . . . . . . 19

4.3.1 Directory hierarchy . . . . . . . . . . . . . . . . . . . . . . . 204.4 Code used from external sources . . . . . . . . . . . . . . . . . . . . 21

4.4.1 Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214.4.2 HTML and JavaScript . . . . . . . . . . . . . . . . . . . . . 234.4.3 Other software . . . . . . . . . . . . . . . . . . . . . . . . . 23

4.5 Back-end design of Krang . . . . . . . . . . . . . . . . . . . . . . . 244.5.1 Database model . . . . . . . . . . . . . . . . . . . . . . . . . 254.5.2 Ontology model . . . . . . . . . . . . . . . . . . . . . . . . 294.5.3 Face detection and recognition . . . . . . . . . . . . . . . . . 364.5.4 Foursquare . . . . . . . . . . . . . . . . . . . . . . . . . . . 394.5.5 Google Calendar . . . . . . . . . . . . . . . . . . . . . . . . 404.5.6 Åbo Akademi Department of Information Technology . . . . 41

4.6 Front-end design of Krang . . . . . . . . . . . . . . . . . . . . . . . 444.6.1 Profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444.6.2 Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

4.7 Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 484.7.1 Complementing location information . . . . . . . . . . . . . 484.7.2 Event notification system . . . . . . . . . . . . . . . . . . . . 504.7.3 Information display with user data . . . . . . . . . . . . . . . 52

4.8 Web SIB Explorer and Smart-M3 Tutorial . . . . . . . . . . . . . . . 53

5 Discussion 555.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 555.2 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

Bibliography 57

Swedish summary 62

Appendices 63

A Configurations 64A.1 uWSGI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64A.2 Nginx . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

B Web scraping 66B.1 research.it.abo.fi login . . . . . . . . . . . . . . . . . . . . . . . . . . 66

iii

LIST OF FIGURES

2.1 RFID tag used at 25C3 CCC conference [1]. . . . . . . . . . . . . . . 7

4.1 Back-end servers and services with corresponding Django apps. . . . 204.2 Complete database model, without database fields. . . . . . . . . . . 254.3 Core app database model. . . . . . . . . . . . . . . . . . . . . . . . . 264.4 Gcal app database model. . . . . . . . . . . . . . . . . . . . . . . . . 274.5 Foursquare app database model. . . . . . . . . . . . . . . . . . . . . 274.6 Freqo app database model. . . . . . . . . . . . . . . . . . . . . . . . 284.7 Krangs ontology classes and their relationships. . . . . . . . . . . . . 314.8 Person class, with object and data properties. . . . . . . . . . . . . . 314.9 Room class, with object and data properties. . . . . . . . . . . . . . . 324.10 Floor class, with object and data properties. . . . . . . . . . . . . . . 324.11 Building class, with object and data properties. . . . . . . . . . . . . 324.12 Phone class, with object and data properties. . . . . . . . . . . . . . . 334.13 Camera class, with object and data properties. . . . . . . . . . . . . . 334.14 VideoData class, with object and data properties. . . . . . . . . . . . 334.15 GoogleCalendarEvent class, with object and data properties. . . . . . 344.16 FoursquareCheckin class, with object and data properties. . . . . . . . 344.17 UserLocation class, with object and data properties. . . . . . . . . . . 344.18 CameraLocation class, with object and data properties. . . . . . . . . 354.19 FoursquareAccount class, with object and data properties. . . . . . . . 354.20 GoogleCalendarAccount class, with object and data properties. . . . . 354.21 Original Local Binary Pattern description [2]. . . . . . . . . . . . . . 374.22 Circular Local Binary Pattern description [3]. . . . . . . . . . . . . . 384.23 Flow diagram for OAuth 2.0 [4]. . . . . . . . . . . . . . . . . . . . . 394.24 Flow diagram for Foursquare Real-Time API. . . . . . . . . . . . . . 404.25 Door displays in the ICT building. . . . . . . . . . . . . . . . . . . . 424.26 User profile page in Krang with functional elements marked. . . . . . 454.27 User settings page in Krang. . . . . . . . . . . . . . . . . . . . . . . 474.28 Flowchart of complementing location information use case. . . . . . . 484.29 Flowchart of event notification Celery task. . . . . . . . . . . . . . . 504.30 Flowchart of a Celery task to change a status display using multiple

data sources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

iv

4.31 Web SIB Explorer Class Tree view. . . . . . . . . . . . . . . . . . . . 54

v

1 INTRODUCTION

Smart-spaces and environments are becoming more popular as we see a growth inthe use of mobile devices and sensors. And with a new interest in the Internet ofthings, we might soon see everyday objects such as chairs and door talking to eachother. The more data are retrieved by sensors the more data storage and processingneeded to get the most out of the information. Data processing however is mostlydone by the owner of the data, and combining data and data mining is becoming thenatural next step to create more intelligent software. In this thesis we introduce away of collecting information from several data sources that feature location data. Bycombining information from these sources we create an application which can trackpeople’s current location with more precision than by using a single source of locationdata.

1.1 AMEBA project

This thesis was done within the AMEBA (Agent based Management of EmBeddeddAta reservers) project. The AMEBA project aim to show the advantages of three-dimensional data storage by using distributed agent-based data management. Thisthesis aims to create a demonstrator for showing some of the features that can be ac-complished by using such data management.

1.2 Problem statement

As storage grows bigger and cheaper, companies are starting to store more and moreinformation than what was possible before. Most of the additional information is nolonger created by humans, but by machines. Information such as logs, meta-data andsensory information are being added to content created by human, and is putting ahigher demand on storage than ever before. However one of the problems with the

1

large amounts of data is that it is never processed in any way. Every time users uploadan image taken by their mobile phone to a photo sharing application it may contain alot more information than just the image. If the GPS was enabled, location informa-tion might be embedded in the image, and other meta-data such as ISO and exposureinformation might be included as well. Facebook also adds information about a userslocation to status update, if the user has allowed it. All of this extra information is onlyused if the service storing deemed it fit for processing. Otherwise they are stored, andmost likely not used until they ultimately are deleted.

We propose a system letting a third party process the meta information, in our caselocation information. By combining location data from several sources, more intelli-gent people tracking is possible which could include predictions of peoples next move.The data can also come from sources such as video surveillance systems, in whichseveral terabytes can be stored every day. The data from such systems might stay un-touched as surveillance system operators might never need access to it. We propose anapplication processing the video stream and extracting addition information, such ashuman recognition and movement prediction. By running face recognition algorithmson the video stream, people tracking could be done automatically. Knowing the direc-tion the camera is pointed in and tracking at what direction a person is walking, onecould follow a person around without human interaction.

The databases storing data in larger applications are very non interactive as pro-grams often have to query information, before they can get data to process. If applica-tions are to respond quickly to new information added, it will have to constantly pollfor the information, which increases the load on the database server. The solution wepurpose to lowering the response time is to use the Smart-M3 triple-store database,which supports subscriptions and can push information to an application when the ap-propriate data has been added to the database. Something which puts less strain on thedatabase, and makes response times smaller.

1.3 Thesis structure

Chapter 2 includes current techniques used for tracking people, as well as the intru-siveness of tracking people with such methods. We distinguished four main methodswhich are used for tracking. Human recognition, which includes using computers forrecognizing people. Electronic tracking, which uses electronic devices such as GPS for

2

tracking. Location prediction, which can be used when enough data exists and lastlyweb services which are the newcomer when it comes to tracking, and offers severallocation based services for tracking people.

Chapter 3 describes how we chose each technology in the demonstrator, and howa combination of them can be made as well as how they complement each other. Wefirst describe the web platform used in our application and how we can connect onlineservices and their APIs to the application. We also show how we can store data fromonline services as well as other data sources. Lastly we describe how we chose theface recognition algorithm used in our implementation.

Chapter 4 contains the implementation of the application. We describe the structureof the project, third party softwares that was used, functionality of the application, andhow all of the components were implemented. Furthermore we describe how the front-end was design as well as give three use cases of our application.

Chapter 5 concludes the thesis with a discussion regarding the demonstrator andits usage. We also discuss future work that could be done to create an even morefeature-rich application.

3

2 PEOPLE TRACKING

People tracking is becoming more commonly used by governments, in health careand by other employers to track workers, perpetrators, patiens or suspicious behavior.This thesis does not intended to cover full-fledged security systems, but it is howevervaluable to consider the many ways of how such systems track people. While it isusually useful to have a human monitoring security systems, in this chapter we willmainly look into automated solutions for tracking. We will go through the advantagesand disadvantages of some of the most common ways of tracking people, includinghuman recognition, electronic tracking, location prediction and web usage. Some ofthese methods have been used for a long time, while others are emerging as new waysof tracking people’s position.

People usually value their privacy fairly high, and many mind having their picturetaken and other people following them. For this reason we will also look into theethics and intrusiveness of tracking people. We also shortly cover ignorance peoplehave towards tracking systems, and why many do not mind people tracking them whenthe method of tracking is less intrusive.

2.1 Tracking techniques

We often refer to people tracking as following people on security cameras and bycross-referencing the cameras location to the human in the video feed you get the per-sons location. This is however only one of several possible ways of tracking. Thanksto key-card systems, mobile location services and Internet usage we can today get afairly accurate position of a person without being in sight. Airports for example are afrequent user of tracking techniques. Airports usually have systems in place to trackpassengers from when they first enter the terminal to when they enter the airplane.Today’s bigger airports will most likely implement technology such as face recogni-tion [5], RFID tracking [6] as well as their normal video surveillance system which

4

has people constantly monitoring others whereabouts. If tickets are booked online,the booking system could get your current location at the time of booking, and have arelatively precise location of the passenger.

The following section will give an overview of some of the techniques used fortracking people. Most of the techniques described below are already in use today andsome are also used in the demonstrator introduced later on in this thesis.

2.1.1 Human recognition

Several techniques for human recognition have been used for numerous years, includ-ing biometric solutions such as iris and retina scanners [7], finger and hand-print scan-ners [8]. As computers have become faster, recognizing people in videos has become alarge area of research [9][10][11]. Lately even DNA is used as a fast way to recognizepeople [12][13]. These ways of recognizing people are used for authentication as wellas security systems where tracking people can be of interest. For instance, finger printscanners has been an optional accessory for computer for some years, mainly to workalongside passwords for authenticating when people log into their operating systemsor applications [14]. In some cases however, this has not been a feasible solution, andhas rendered the systems they are used on even more vulnerable than before the newsecurity measures were installed. The problems with fingerprint scanners are usuallynot the hardware itself, but the underlying software using the hardware [15].

Recognizing people in videos and pictures is a hard task, which has led to thedevelopment of an extensive amount of algorithms produced and tested for this purpose[16]. Most of the solutions to date use a combination of already existing techniques inimage manipulation and combine them into new solution, as can be seen with PrincipalComponent Analysis (PCA) techniques, which uses eigenvectors to find and recognizefaces [11]. While the main focus of this project is to implement a facial recognitionalgorithm into the program, several other ways of recognizing people in videos exist.For instance the posture of a person can be used to determine if a human is in a video[17], a technique which could be combined with algorithms recognizing movementto create a system to track people [18]. Some techniques for recognizing people usethe context of which people are in. For instance if there is a photo in a newspaper, acomputer can based upon the caption determine who is in the picture. In this case, acombination of face detection and language models is used to recognize the people inthe image [19].

5

A persons interest may also be of value. With the launch of the Microsoft Kinect[20], researchers could acquire on a relatively cheap depth camera. With depth infor-mation at hand, several new solutions for activity and face recognition could be made.Solutions such as 3D face reconstruction became cheaply available [21], and new facerecognition algorithms that benefit from depth information was developed [22]. It alsobecame easier to distinguish between gestures, which in turn made it easier to de-termine the activity of a person [23]. While activity recognition algorithms alreadyexisted before the Kinect [24], research benefited greatly from it. With the help of a3D image, robots could also be made to easier distinguish between people, and mademore autonomous by being able to see image depth [25].

2.1.2 Electronic tracking

The use of external devices can also be used for tracking people. An estimation of thelocation can be acquired by tracking their use of key-cards or passports. There are alsoother techniques that can be used. Bluetooth tracking can be used if a user has lefttheir device in discoverable mode and Wi-Fi MAC addresses can be tracked when auser connect to a new access point [26].

As Google generated their street-view service, they collected the MAC addressesof most Wi-Fi access points surrounding their camera cars [27]. By doing another driveby, and by finding the same MAC address they could know if you have moved or not.There is no information pointing to Google actually doing this, but as more devicescomes integrated with wireless technologies, tracking people will become easier if nocountermeasures are put in place.

Combining technologies to get a better accuracy of a person’s location can alsobe used as Mori et al. present by combining pressure sensors in the floor, with RFIDtracking to get a good estimate of a person’s location [28]. They did so by sensing thepresence of a human with the presence sensor, and identifying the person by triangulat-ing the position with multiple RFID transceivers. There are also open source solutionsfor RFID tracking, such as the one developed by the OpenBeacon project [1]. TheirRFID tags and readers have been used to track the participants of conferences such as25C3 CCC and BruCON 2011 [1].

GPS tracking systems are also an extensively used method of tracking, as it givesa good outdoor position location estimate, and is constantly getting better at indoor

6

Figure 2.1: RFID tag used at 25C3 CCC conference [1].

locationing. GPS tracking systems is also a relatively cheap method of tracking as noexternal sensors has to be put in place by its operators. This also makes it suitable forlong range tracking, such as of people traveling by car, Alzheimer’s patients or peoplewith dementia wandering away from caring facilities [29] [30].

2.1.3 Location prediction and data mining

With enough data about a persons whereabouts, one could eventually predict theirnext move with a good precision. As shown by Ingrid Burbey at Virginia PolytechnicInstitute and State University, where a person’s location could be estimated with aprediction rate of 77%-91% with an allowed tolerance of plus/minus twenty minutes[31]. In this case, WiFi access points (APs) were used to track college freshmen for14 weeks, which gave the initial test data. Burbey then used Markov models on thedata to predict the user’s next location. Similar predictions could also be done usingcellular networks, which could enable mobile operators to predict where you will begoing next [32].

The winners of Nokia’s Mobile Data Challenge 2012 also showed that by com-bining a user’s location with the location of his phone book contacts, a good locationestimate could be achieved [33]. Their algorithms could estimate a person’s location24 hours into the future with an average margin of only twenty meters. Solutionssuch as this could be used by the authorities to estimate when a crime is about to becommitted. Commercial applications could also benefit from location prediction when

7

showing advertisements or giving suggestions to points of interest that might interestthe consumer.

When using location prediction the future location of a person is not always ofimportance, but rather knowing points of interest connected to the person. Researchersfrom Georgia Institute of Technology shows that by tracking a person’s movement witha GPS, and feeding the information into a Markov Model, they could estimate user’smain points of interest [34].

2.1.4 Web services

Services also exists which lets its users willingly share their locations with other peo-ple. Services such as Foursquare and Google Latitude are used mainly to share a user’slocation, while other services such as Facebook and Google+ implement a locationsharing features only as a part of their services.

Foursquare allows its user to check into places while competing for points withyour friends. The more places the user checks in, the more points that person gets.It also helps keep track of where a user friend is, and help people to find each other.Google Latitude works by sharing its user’s current location with friends. The usercan choose how exact the position is, by selecting if the location should be providedas an exact GPS coordinate, or on city level precision. Facebook and Google+ sharethe location of the user when a new status update is created. This is an opt-in featurewhich both services provide. This gives the users status update more context.

While these services do not aim to track people per se, when combined, the amountof information being shared could be used to track a user’s whereabouts, and might alsogive a prediction of the user’s next location.

2.2 Ethics, intrusiveness and ignorance

While tracking people is a great asset to a security system, or even to some companiesthat want to streamline the work more efficiently, ethics and security might be an is-sue. People might not appreciate the spreading or private information, which peopletracking gives arise to.

Norwich Union, now part of Aviva, was a British insurance company that issuedGPS tracking devices on cars registered by drivers between the age of 18 and 21,charging premiums depending on the time they used their vehicles. Disney has offered

8

GPS tracking for parents to put on their children, in order to know their location at evertime. Some truck driving companies in the USA also issued GPS tracking to reducethe amount of overtime, by tracking the movement of the trucks [30]. Systems likethese might have moral or ethical issues, as the people being subjects to the trackingmight be able to choose whether or not tracking is enabled. It might be that the personis underage, or that the work place simply demands that GPS tracking systems areinstalled in work related vehicles. Whether or not it is morally right to track people is avery subjective topic. Some people might not mind being tracked, while others beliefsor ethics might make them of a completely different opinion.

On of the simple solutions of figuring the feasibility of a tracking system is tocollect the opinions of people being exposed to the technology [35]. The advantagesand disadvantages should also be weighed against each other, as people usually wantto stay private, and not have other people tracking them.

During the time of this project there was a camera set up in the corridor of thelab for 6 weeks, mainly to test facial recognition algorithms. The camera was locatedclose to the ceiling and was barely visible. An e-mail was sent to the workers of thelab, stating that a camera had been put in place, and if anyone feels uncomfortablewith it being installed they should let us know. While complains were given, one couldeasily note that people were looking straight into the camera when they walked by.This behavior did eventually stop after some time, which begs the question why? Themain answer from people working in the lab was that they forgot that the camera waseven there. While the psychology behind this is not going to be covered by this thesis,it is still interesting to note how people tend to ignore they are being monitored after awhile.

9

3 CHOOSING AND COMBININGTECHNOLOGIES

In this chapter we present the technologies used in our implementation as well as com-parisons to similar solutions. The main parts of the system is the application plat-form, application services and the data storage. These represent respectively the sys-tem which the program will run on, the services the application uses as well as wherethe data is stored during the usage of the program.

As stated in Section 1.2 the aim of this project was to combine several of exist-ing technologies into one single system. We chose the technologies in part by usingsoftware created under the DIEM research project and in part by technologies chosenat the planning stage of the thesis. The part included from the DIEM was the Smart-M3, a smart-space solution, which included a triple store database. Storage solutionsfor image and video files were also compared, as well as creating native against webdevelopment approaches.

As the system should include several ways to track a person, facial recognitionalgorithms and web application programming interfaces (APIs) were implemented tocreate a diversity of technologies.

3.1 Application platform

When choosing the application platform for a project of this nature we had two choices,we could create a native application created specifically for one platform, or a web ap-plication. While native applications run on the user’s operating system which offersspeed and works offline, web application offer device versatility and robustness whendeploying. Both have their benefits, but as we had more experience with web devel-opment, a web platform was chosen. We will discuss our choices of software andintegration in the following sections.

10

3.1.1 Web framework

The framework chosen for our application was Django [36], an open-source Pythonweb framework. The main reason for choosing Django was our precious knowledge ofthe software, which made development faster and the learning curve smaller. Djangois one of the largest web frameworks for Python, and has been in development since2005. It follows the mode-view-controller architecture, which separates the handlingof the data with users interacting with it. One of Django’s main advantages is itsability to reuse code, due to its use of apps. Apps are parts of a program, created fora specific task. Django also uses an object relational mapper between data modelswritten as Python classes and a relational database. It uses a web template system,Jinja, for handling information between the database and the user. Django also has aURL dispatcher, programmable by regular expressions.

As Django uses apps to separate different tasks, it makes the task of further ex-tending and existing application easier. Apps have their own models, views, URLdispatcher, tests and can be included in an existing program by copying the app folderinto the project. Linking a new app with the rest of the application is made by modify-ing the main URL dispatcher to include the new apps dispatcher.

Django also comes integrated with an admin interface, where one can control thedatabase content. Admin interfaces usually have to be created by the application devel-opers, but as Django comes bundled with one, it speeds up the development process.The admin interface can also be modifies by developers, by adding features and chang-ing styling.

3.1.2 Web application and background tasks

Web frameworks such as Django do not support native background processing, andother applications have to be linked with Django to enable such functionality. WhilePython modules such as subprocess may be used to run processes in the background,a more robust solution was to use a task queue such as Celery [37]. Celery is an asyn-chronous distributed task queue written in Python. It uses a broker to store new tasksand handle results; the recommended broker by the Celery developers is RabbitMQ[38].

For running background tasks in Django with Celery, the Python package django-

celery has to be present, and configurations has to be made to the Django applications

11

settings file.When using the RabbitMQ broker, only a minimal configurations has tobe made to Django, which includes small modifications to the applications settings.py

files. When creating tasks, a task file which includes functions written to be run inthe background has to be created. These functions can then be called by another ap-plications, or run as a scheduled task. Django, Celery and RabbitMQ can also run onseparate hardware if necessary. This distributes the work load, which makes the appli-cation more robust. As this is a small application we chose to keep all servers runningon the same machine.

3.1.3 Web server

Django comes bundled with a development server. The Django Software Foundationhowever does not recommending the usage of this server for deploying applications tothe public. There is however several choices when choosing a server to run Djangoapplication on and some performs better at serving particular sets of data. This led usto use a combination of servers to host our application. As Django is written in Pythonthe first choice was to select a Web Server Gateway Interface (WSGI), which willserve as an interface between the web server and our application, sometimes calledan application server. We also had to choose a web server to make the applicationavailable on the network.

The web server of choice when running web based Python applications is Nginx.It is a web server and reverse proxy written in C. According to the Internet servicecompany Netcraft, Nginx is one of the fastest growing web servers on the Internet, witha current market share of 12.07% [39]. The main advantage from other web serverssuch as Apache is speed. Several benchmarks show that Nginx is the fastest web serverto date [40][41][42], Nginx has also proven to be fast at serving static content, whileletting other processes handle dynamic processing. It also has good integration withGunicorn and uWSGI which was the reason we chose Nginx as our web server

From previous experiences we knew that the two main choices for WSGI serverswere uWSGI[43] and Gunicorn[44]. Both are pre-forking WSGI server, which meansthat they spawn worker nodes at start-up, cutting the overhead of forking when runninga new task. The server was selected based upon favored features such as ease of use,speed and stability.

Gunicorn is written in Python and is created to rely on a proxy server to handlethe first steps of HTTP requests. This means that a web server would receive HTTP

12

requests, and if the requests are dynamic it would send them to Gunicorn, which pro-cesses them and return them to the web server once finished. Gunicorn can be usedas a web server directly, but as response times would be an issue, and it is not devel-oped to handle static files, such as images and cascade style sheets (CSS), its develop-ers do not recommend that setup [45]. There are several ways to integrate Gunicornwith Django, Django applications can be run directly with Gunicorn using the guni-corn_django command and add parameters, or use a configurations file and load it viathe gunicorn_paster command.

uWSGI is written in C, and similarly to Gunicorn it relies on a proxy server tohandle the first step of client requests. Its main method of configuration is using an inifile. The main advantage from Gunicorn is that uWSGI is better integrated with theweb server Nginx. This makes the connection between Nginx and uWSGI faster thanthe one between Gunicorn and Nginx. This is also the main reason we chose to useNginx and uWSGI for deploying our application.

For combining uWSGI and Nginx with our Django application we first created aconfiguration file specifying project specific paths, and uWSGI settings. We configureduWSGI to use a unix socket instead of a TCP socket as this is usually the fastestconfiguration when running uWSGI and Nginx on the same server. A PID file wasalso specified for easy access to the process ID as can be seen in Appendix A.1. Weconfigured Nginx to connect to the socket created by uWSGI and also configured itto server all static files, while uWSGI handled the dynamic processing. Nginx wasalso configured to run over SSL to make all connection encrypted as can be seen inAppendix A.2.

3.2 Online APIs

One of the main purposes of the application was to be able to track people’s where-abouts, which made online services supporting user location of the particular interest.We found that Facebook, Foursquare and Google Calendar were good sources of suchlocation information. As our application was meant as a proof of concept applica-tion and integrating Facebook would take too long we decided to use Foursquare andGoogle Calendar as our main sources of information.

Foursquare uses oAuth for authentication and API requests are handles over a se-cure SSL connections. They also offer a push API, which sends information to a

13

developer set URI, every time an authorized user with ones application checks in. Thislightens the load of the application, as it does not have to poll for new check in man-ually. We chose to use the push API as it could be integrated with the applicationeasily.

The push API works by sending JSON data to a specified address, which the de-veloper then needs to process. One way to handle the data is to store it in a database,and use upon demand. Our solution stores the check-in information in the Smart-M3database every time a user checks in a location. This way, applications that want tointegrate with our Smart-M3 SIB, is able to get a notification when a new check-in hasbeen made, as well as when the user’s location is changed.

3.3 Databases and media storage

Data storage is one of the main parts of web applications and they are constantly hit byusers accessing data when new web pages are loaded. As one of the project require-ments was to use the Smart-M3 software, we began exploring its possibilities, and howto integrate it in to the application. A comparison between multimedia storage solutionwas also made, comparing databases to file systems.

3.3.1 Smart-M3

Smart-M3 is an open-source software that aims to increase the intelligence of entitiesand devices by connecting them together via the sharing of data. Smart-M3 strives tobe multi-vendor, multi-device and multi-part compatible, therefore the M3 at the endof the name. Smart-M3 uses Semantic Information Brokers (SIBs) to distribute thedata between devices. A SIB is placed on top of a Triplestore database, and acts as ahandler in between the actual database and the Knowledge Processors (KPs), as can beseen in Figure 2.1. Several SIBs can be connected to each other, to form a smart space.The SIB is not tied to any specific triplestore, but as of writing there are only twopublicly available implementations in the Smart-M3 Sourceforge repository [?]. Oneid developed by Nokia and the other by The Advanced Research Center on ElectronicSystems at the University of Bologna.

What distinguishes Smart-M3 aside other triple and graph storage solutions is itsway to handle the flow of data in the network. Instead of constantly querying for newdata on the triplestore, devices can set up subscriptions to certain queries and get a

14

push like behavior from the system when the subscribed data is modified. The workernodes in a Smart-M3 smart space are called Knowledge Procesors. The KPs handle alltransaction with the SIB such as insert, update and delete actions. The KPs also handlesubscriptions to the SIB.

It should be noted that SIBs and the KPs can run on the same machine, but usuallythe SIB runs on a computer faster and with more memory then the rest of the KP nodesin the network. KPs are also made to be multi-platform, and several implementationsexists including versions for C [7] , C# [8], Python [9], Java [10], JavaScript [11] andPHP [11].

3.3.2 SQL and NoSQL combination

For database storage we chose to combine a NoSQL database [46], Smart-M3, with aPostgeSQL database[47]. The choise for Smart-M3 came from the project specifica-tion, and PostgreSQL has proven to work well as a database server for web projectsin the past. The reason to combine these databases instead of using just one was be-cause Smart-M3 does not support user authentication, which makes it a bad choice forstoring passwords and other sensitive information. The PostgreSQL also serves as aredundant database while moving over part of the database handling to the Smart-M3.We combined these two databases by overriding Djangos save method when savingmodels. By doing so we could easily integrate Smart-M3 while still holding on to thePostgreSQL database for redundancy.

A benefit of using Smart-M3 is also that other services can connect to the SIBdatabase and create applications based on the information flow in the SIB. As Smart-M3 offers the possibility of subscribing to information, other applications can workwith the data of another application, without knowing how that information came to bein the first place. Data can also be fed back into the SIB, to complement or changingthe information of another program. This way applications have only one source ofinformation, and does not have to communicate directly with each other. This reducesapplication coupling, and as several SIBs can be connected to each other there willalso be redundancy, this making applications more robust. All applications connectedto the Smart-M3 smart-space should however be able to handle changes from anotherapplication, without failing. This making the development process a bit more tedious.

15

3.3.3 Image and video storage

For media storage one of the simples and most effective solutions is to store videosand image on the file system, while keeping track of their file paths and meta-data ina database. While using only a database offers some good features as ease of syn-chronization between databases, easy access and less fragmentation. File systems areusually faster, takes load of the database, requires less code to access its files and givesbetter control over media files. Storing files on the file system also simplifies the pro-cess of exporting files to other programs for transcoding or other post processing work.

When choosing a good file system to store and accessing large amount of fileswe compared EXT4 and XFS. EXT4 is the standard file system use in many mod-ern day Linux distributions such as Ubuntu and Fedora and XFS is promoted as ahigh-performance, highly scalable file system for large data storage. Empirical testshowever show that the difference between the two file systems is minimal, and thatthere are no great benefits of using XFS over EXT4 [48]. The extra time of setting upan XFS file system is therefore not beneficial, and we thereby chose to use EXT4.

3.4 Face detection and recognition

Face detection and recognition has existed for a long time, and several algorithms havebeen developed since Woodrow W. Bledsoe first showed that face recognition could bedone with computers in the early 1960s [49][50]. As we had set out to use the OpenCVlibrary for image processing, we found that it offered one ready-made solution for facedetection. We found that the most research regarding face recognition had been doneusing Eigenfaces, Fisherfaces as well as Local Binary Patterns (LBP).

For face detection, OpenCV uses Haar Cascade Classifiers, which work by usinga cascade file, containing Haar features of faces which are compared to the featuresof an image to determine the existence of a face [51]. A Haar feature is calculated byfirst placing simple rectangular sections, containing two or more adjacent rectangularregions which are either white or black over a detection window. The pixel intensityinside each region is then summed up. The regions summed up intensities are thenused to categorize section of the window. If these sections exist in an image, we havea match in the cascade file. This technique requires the developer to train and createcascade files in advance, which includes the Haar features of the object to be detected,in our case the face.

16

For face recognition we compared research done using Eigenfaces, Fisherfaces andLBP. Eigenfaces uses principal component analysis (PCA) to extract facial featuresfrom a set of images of a person [52]. Fisherfaces on the other hand uses linear dis-criminant analysis (LDA) to extract facial features [53]. Both methods need trainingdata of people to be able to later recognize them. The training data usually consistsof several pictures of the subject, in various lightning conditions. These two methodswork fairly well, but both algorithms have problems when lighting conditions changesfrom the initial training data. Local binary patterns however are less error prone whenit comes to recognizing faces in new lighting conditions [9]. It also needs the leastamount of training images of the three, to get a fairly good prediction rate. This wasthe main reasons for choosing to use LBP as our face recognition algorithm in thisproject. In subsection 4.5.3 in chapter 4 we will go into greater details about how LBPworks, and how it was implemented into the project.

17

4 IMPLEMENTATION

This chapter covers the Krang demonstrator created for the AMEBA project. It presentsthe implementation of both the front- and back-end implementation. The implementa-tion is structured in part by Djangos app structure and part by best practice implemen-tations from previous knowledge. The front-end was developed with usability in mind,as the demonstrator was to be used as a showcase for the AMEBA project. Use casescenarios will be described in section 4.7 at the end of this chapter.

4.1 Platform

The back-end implementation was created for the x86 and x64 instruction set plat-forms, running a Linux based operating system. The implementation has howeverbeen tested to work on a Raspberry Pi which uses an ARM1176 processor using theARMv6 instruction set. The front-end implementation was created for non-mobileweb browsers utilizing the Gecko or WebKit layout engine such as Mozilla Firefox,Google Chrome and Apple Safari browsers.

4.2 Krang Demonstrator

The Krang demonstrator is a distributed web-based people tracking application. It wascreated for the AMEBA project, as a proof of concept for demonstrating smart-spacesand intelligent software. A smart-space can be a specific location on environmentwhich uses electronics, such as computers and sensors, to give more information aboutthat particular space. It can also include data from existing services, which are inter-preted in a new way, to get a more intelligent environment.

The main purpose of the demonstrator was to use already existing software andalgorithms to create a program which could benefit from having several sources of

18

information. We focused mainly on people tracking and face recognition. By combin-ing services such as Google services, Foursquare and face recognition, we created aprogram which could track a person’s location and use the information to notify otherusers of what people are currently doing. The demonstrator integrates the Smart-M3triples store which enables other devices to access the information created by our soft-ware. This makes the software application more modular, and creating new featuresfor the program can be done by only managing the Smart-M3 data. Therefore it doesnot have to be integrated into the demonstrator itself.

Another purpose of the demonstrator is to distribute the processing as much aspossible, to make it mappable on several processing elements, or devices. For thisreason the servers and middleware used in Krang was partly chosen by how well theyare able to scale.

4.2.1 Functionality

The Krang demonstrator is able to tell the location of a person, based on the datasupplied from multiple sources. The source being either user input, data received inrelation to user actions or data retrieved automatically through methods such as facerecognition. The location is then used to determine if a user is currently available inhis room, close by or at a different location. After determining the users availabilityother devices such as lights or status updates can be set accordingly.

A calendar late-to-event feature is also implemented, which is able to tell if a personis going to be late to a meeting. By checking for a user’s current location and currentcalendar events, we can tell if a person is going to be late to a meeting, and is then ableto notify the other participants by e-mail. This particular scenario is discussed in detailin the event notification system use case in section 4.7.2.

4.3 Implementation overview

The software is created using the Django web framework, and uses its app structure forseparating different tasks. As a rational database, we used SQLite during developmentand PostqreSQL during production, and the Smart-M3 SIB was used for storing triples.The database models and their dependencies can be seen in Figure 4.2 and the ontologyclasses for the triple store can be seen in Figure 4.7.

19

Figure 4.1: Back-end servers and services with corresponding Django apps.

The implementations back-end is written mainly in Python, with the use of thirdparty modules, some of which make calls to C or C++ modules. The back-end ishandled by Nginx, uWSGI, Django, Celery, RabbitMQ, OpenCV as well as APIs forGoogle Distance Matrix, Google Calendar and Foursquare. The front-end was createdusing the front-end framework Twitter Boostrap and the javascript library jQuery aswell as custom written jinja2 HTML templates, JavaScript and CSS files. The differentback-end apps and their dependencies can be seen in Figure 4.1.

4.3.1 Directory hierarchy

The application is structured into several directories, some with subdirectories. Thismakes the application structure more clear and development easier. There is no clearstructure guideline from the Django Software Foundation how applications should bestructured other than placing apps in separate folders. The following structure wasused for the implementation:

krang/ This folder is the main project folder of the Django application.

/core/ Core app of the application. Includes the main functions for handlingwhat is shown to users.

/templatetags Template tags related to the core view.

/foursquare/ Foursquare app, holds functions related to handling Foursquarerequests and settings.

/freqo/ Face recognition app.

/gcal/ Google calendar app, holds functions for handling Google Calendar re-quests and settings.

20

/templatetags Template tags related to the google calendar view.

/libs/ Libraries developed specifically for this application, as well as third partylibraries used, such as face recognition algorithms.

/facedet Face detection library.

/facerec Face recognition library.

/helper Helper functions for the face recognition library.

/videofacerec Library for linking face detection and recognition.

/helper Helper functions for videofacerec library.

/logs/ Holds main logs for the application.

/media/ Static media files, developed for the application as well as third partylibraries and frameworks.

/css Cascade style sheets related to the project.

/vendor Third party CSS files.

/img Images related to the project. Subfolders might be dynamically gen-erated by the application.

/js JavaScript files related to the project.

/vendor Third party JavaScript files.

/static/ Static files for the Django admin interface.

/templates/ Application templates

/core Templates for the core view.

/templatetags Core app templates spcifically for the templatetags.

4.4 Code used from external sources

The following section contains code from external sources that was used in our appli-cation. It includes several Python packages, front-end web frameworks and librariesfor computer vision and background processing in Django.

4.4.1 Python

Several Python packages were used for developing or application. It should be notedthat some of the packages listed below have their dependencies not listed. There were

21

also packages, helping the development of the application that were used, but not listedhere. Their packages can be found in the requirements.py file in the applications rootfolder.

Mechanize Mechanize is a Python port of a popular Perl module, for stateful pro-grammatic web browsing. It uses urllib2 for fetching web pages, and can also be usedas an FTP client. It includes features such as cookie handling, HTTP refresh modifi-cation, robot.txt handling, proxies and HTTP authentication. Its main use is to writeautomated web crawlers or screen scrapers.Licence: BSD

BeautifulSoup BeautifulSoup is a tool to help with screen scraping web pages inPython. It turns HTML or similar code as XML into a Python object, which offersfunctions for searching, navigating and modifying the code. In combination with pack-ages such as Mechanize, Beautiful Soup offers an easy way of screen scraping.Licence: BSD

Celery Celery is an asynchronous distributed task queue; its main focus is real-timeoperations but can also be used for running scheduled tasks. It uses a message brokerfor message handling, which needs to be installed separately.Licence: BSD

Django Django is a web framework for Python that implements a model-view-controllerarchitecture. It includes an object relational mapper that handles the communicationPython classes to database models. It offers support for web forms, authentication,caching, testing interfaces and several other features. It also includes an admin inter-face for handling database entries. Django integrates a development server, but shouldbe run on top of a web server such as Nginx using WSGI when in production.Licence: BSD

Google APIs Client Library for Python Googles API Client Library offers accessto most of Googles APIs using Python calls. In addition it also offers an OAuth2module, which can be used with any OAuth2 server. In this project the library wasused for accessing the Google Calendar API.Licence: Apache License 2.0

22

OAuth2Client OAuth2Client is part of Google’s APIs Client Library for Python.OAuth2Client is an OAuth2 client which enables Python calls to be made directly toan OAuth server, such as Googles, Facebooks or Foursquares OAuth API. It hides allHTTP calls so that all calls can be made in a more Pythonic way. It also offers ready-made made model classes for Django, which can be used for storing user credentials.Licence: Apache License 2.0

Flask Flask is a lightweight web framework for Python. It uses Jinja template en-gine, and offers a build in web server. Flask relies on extensions for features such asdata base handling, web forms and authentication.Licence:

4.4.2 HTML and JavaScript

Twitter Boostrap Twitter Boostrap is a front-end framework for web development,developed by Twitter. It offers HTML and CSS templates as well as several Javascriptextensions.Licence: Apache License v2.0

jQuery jQuery is a client side JavaScript library. It is the most used JavaScript li-brary today [54]. It includes wide range of features such as Document Object Model(DOM) traversal and manipulation, AJAX as well as being multi-browser compatible.Licence: MIT

CodeMirror CodeMirror is a in browser code editor written in JavaScript. It sup-ports syntax highlighting for a wide range of programming languages such as C,Python, SQL, SPARQL and CoffeeScript. It is customizable via CSS and supportsa JavaScript API.Licence: MIT

4.4.3 Other software

RabbitMQ RabbitMQ is a message broker that can be used in combination withCelery. It stores all jobs and their results for Celery tasks, and can run a separate server

23

than Celery if need be.Licence: Mozilla Public License (MPL)

OpenCV OpenCV (Open Source Computer Vision) is a computer vision librarymainly focusing on real-time processing. It is developed by Intel, but can be run onseveral architectures such as x86, x64 and ARM. It is written in C++ but has wrappersfor languages such as Python, Java and MATLAB. OpenCV include several computervision algorithms for applications such as face and gesture recognition, robotics andmotions tracking.Licence: BSD

SimpleCV SimpleCV is a computer vision library build upon OpenCV. It offersshortcuts and combines features of OpenCV for faster development. Some examplesbeing Kinect, JPEG stream and macro support. SimpleCV is written in Python.Licence: BSD

Smart-M3 Smart-M3 provides a semantic web information sharing structure by com-bining devices together with a central database. It is multi-platform and includes inter-faces for several programming languages including Python, Java, JavaScript, C, C++and C#.Licence: BSD

4.5 Back-end design of Krang

In this section we discuss how the chosen technologies were implemented and howthey connected to the Django web-framework. Firstly we go through how the databaseand the ontology are structured. Django uses an app-structure, where each app includesa specific feature of a Django application. Every app includes its own database struc-ture, and we describe the app and the database model at the same time, as the modelsdefine the app. Secondly we define the ontology used by the Smart-M3 SIB. We alsodescribe how the ontology and the app models are connected together. Thirdly wedefine how face recognition using a local binary pattern algorithm was implemented.Forthly the social media service Foursquare is then described, and how their real-timeAPI can be used to get push notifications when a user checks-in. We then describe

24

Google Calendar, and how we integrates calendar events in the application. Lastlywe describe how we got information from the Åbo Akademi Department of Informa-tion Technology web pages by web scraping and how the information is used in theapplication.

4.5.1 Database model

Figure 4.2: Complete database model, without database fields.

Django uses an object relational mapper that works between Python classes and re-lational database models. The entire database model can therefore be written in purePython, and Django will then translate it into SQL calls. The developer does howeverstill need to define the type of database fields to use, and properties such as the lengthof a character field. One can also define custom field which makes it easier to add ad-ditional information in comparison to manipulating the database directly. Django alsohandles the connection to the database, and supports several databases, and databasetypes can be change without the need to change the database model.

Every app in a Django application has its own models file, where the models spe-cific for that app specified. When Django synchronizes with the actual database, allmodels files will be read, and the appropriate fields and columns will be created in thedatabase. Figure 4.2 shows the structure of the entire database without the field values.The following sections will cover each of the apps database models separately.

25

Core

Figure 4.3: Core app database model.

The core app consists of three classes. Firstly an extension of Django’s User model,which adds fields for storing additional information about a user. As the core applica-tion includes a screen scraper for getting information from research.it.abo.fi, additionalfields was created to store the information returned by the scraper. This informationis stored in the UserProfile model, which include fields for information such as phonenumbers, information display text, room number and current location. The UserProfilemodel uses a one-to-one field, to extend Django’s User model. This is the recom-mended way to extend the User model by the Django Software Foundation.

Secondly a social profile was also added, for storing information about the user’ssocial media connections. Some fields were created as character fields, to store theusers username, while others, such as personal blogs or the users Google+ page wereadded as URL as we cannot predict these links. The user profile model is linked withthe user profile via the foreign key social in the user profile.

Some of the debug fields are still left in Figure 4.3. The fields temp_pass andcookies are used for debugging certain features, as the program is not yet ready to beused by end users. It should also be noted that the user model has been left untouched,and is developed by the Django developers and not written as part of this project.

26

Google Calendar

Figure 4.4: Gcal app database model.

The Google calendar app uses Googles OAuth2Client package, which includes twospecific Django database fields made for storing OAuth credentials. The fields Flow-

Field and CredentialsField are used for storing information such as the users accesstoken and user id.

As users can have several calendars in the Google Calendar account, a settingsmodel was created for storing the specific calendar the user wants to use. The Settings

model, as well as the FlowModel and CredentialsModel are linked to the User modelvia an id field.

Foursquare

Figure 4.5: Foursquare app database model.

The foursquare app uses three models for storing data. The FSUser model holds infor-mation about the user’s identity, including the user’s access token, which is used forauthenticating the user Foursquare account with our application. The fs_id field holdsthe users unique Foursquare ID, which is used for identifying the user. The Foursuqareuser is also linked with his or her user profile via the user field.

27

A settings model was created for holding Foursquare specific configurations. Thefield only_ict holds information if a Foursquare check-in should only be stored if itis related to the ICT building at Joukahaisengatan 3-5 in Turku. The field no_update

specifies if the users Foursquare check-in should be registered at all. The settingsmodule is linked to a user’s foursquare model, via the settings field in FSUser model.

A user’s Foursquare history is stored in the FSHistory model. It has fields forstoring the check ins location, city, country, coordinates and timestamp. It is linkedwith the user via the user field which is a foreign key to the User model.

It should be noted that Foursquare uses OAuth for authentication as Google Cal-endar. The choice of not using the third party package from Google for authenti-cation with Foursquare was made to better learn how OAuth authentication works.The oauth2client package could be used with the Foursquare model by including thedjango_orm modules FlowField and CredetialsField functions.

Face recognition

Figure 4.6: Freqo app database model.

The face recognition app includes two models for storing user specific data, three mod-els for cameras and their settings and one model for storing a history of when usershave been recognized by the cameras. Some of the fields pictured in Figure 4.6 arenot in active use in the program, and are there for future features. These fields areexplained in the paragraphs below.

The Fuser model stores data about a face recognition user. It has two foreign keys,one to the User model and another to the FSettings model, which holds user specificconfigurations. The face_id field holds an ID for the specific user. At this point in

28

development, the application will store the user’s university login name. The images

field holds the number of available images of the users face.The FSettings model keeps track of the users configurations. It features three alter-

native of which coffee_tracking is currently not in use by the application. The history

fields stores information about if the user want the application to keep a history ofwhen the user has been recognized by a camera. The tracking field holds informationabout where or not the user want to be tracked by cameras. This is to give the user theability temporarily disables tracking by face recognition.

The Camera model stores data about camera locations and gives a description aboutthe cameras. Is has one foreign key, settings, to the CameraSettings model which holdscamera specific configurations. The fields location and location_name differ in that lo-

cation stores an exact location, such as an address or coordinates, while location_name

stores an verbose name of the location. The name filed store the name of the cameraand the field description give a short description of the camera.

The CameraSettings model stores information about IP and path of the video streamin ip and video_path while the source filed stores information about the kind of mediathat can be fetched from the camera. The default choices being video or an image.The resolution field holds information about the resolution of the video stream or theimage and fetch_amount tells how many videos or images should be taken if motionis detected. Every camera can store the media in separate FTP server, and the ftp,ftp_user and ftp_pass fields hold authentication information about the camera specificFTP server. The ftp_wait fields hold information about the timeout value for the FTPconnection.

The history model links users and cameras together by storing information aboutwhen a user has been recognized and by which camera he has been seen. A time-stampkeeps track of when the user was recognized. The coffee field is not in current use inthe application, and was meant to track the amount of times a users goes for coffeeduring a work day.

4.5.2 Ontology model

When working with a graph database, such as the Smart-M3 SIB, an ontology modelgives a structure to the data added to the database. It shows the connection betweendifferent classes, and specifies types for properties. While an ontology is not necessaryfor a graph database, it makes developers job easier as a clear structure of the database

29

is given from the start. The ontology language used in our application is the WebOntology Language (OWL).

We tried to create an ontology that resembled the rational database as close aspossible. Some parts, such as authentication information was left out as the SIB doesnot support user authentication, making it publicly available for everyone who hasnetwork access to it. As the application was meant to run as a distributed system, wecould not assume that the SIB and the other parts were to be run on the same computer,or network. These issues rendered the SIB insecure to store sensitive information.

Ontologies consist of classes, properties, instances and relations, but can also in-clude rules, axioms and restrictions. A class is a type of an object, and all instanceshave a class. A class can be a subclass of another, thereby inheriting all of the parent’scharacteristics. As we are using OWL all classes are subclasses of the root class Thing,which can be seen in Figure 4.7. Properties give classes certain characteristics, and canbe compared to fields in a relational database. There are two kinds of property types,object properties and data properties. Object properties link classes together and dataproperties holds data values for a class. Naming of properties are often done by includ-ing the prefixes has or is followed by an attribute. The attribute for object propertiesare often named by including the class name after the prefix. We will now describe theclasses and their properties used in our application.

30

Figure 4.7: Krangs ontology classes and their relationships.

The ontology is first split up in to a physical and an abstract class, to give a clearseparation between tangible things and data objects. These classes do not have anyproperties, and serve mainly as a separator.

Figure 4.8: Person class, with object and data properties.

Subclasses for PhysicalThing include the classes Person, Environment and Device.The class Person corresponds to the User class in the Django model. A person can belinked to several phones and a user specific Google Calendar, Foursquare, room andlocation class, as pictured in Figure 4.8. It has one data property, hasName, to store

31

the person name.The environment class includes three subclasses and no properties and serves as a

class separator. The environment here specifies the physical environment of a location,it includes the subclasses Room, Floor and Building .

Figure 4.9: Room class, with object and data properties.

The Room class includes one object property, linking a room to a specific floor,and one data property storing the room name or number as pictured in Figure 4.9. Thisclass is used to store a person’s room, and to specify the exact location of a camera.

Figure 4.10: Floor class, with object and data properties.

The Floor class includes one object property linking a floor to a building, and onedata property storing the number or name of a floor, as pictured in Figure 4.10. Thisclass is used mainly to give more information about a location, for instance if a roomnumber is known, the floor number is also known.

Figure 4.11: Building class, with object and data properties.

The Building class includes two object properties for linking floors, and rooms toa specific building as pictured in Figure 4.11. This class is used as an extension to theroom and floor class, to get more information about a location.

For handling devices, the Device class was created as a subclass of PhysicalThing.It has two sub sections, Phone and Camera. It has no properties, and serves as aseparator, and makes querying for all devices easier.

32

Figure 4.12: Phone class, with object and data properties.

The Phone class includes one object property linking a person and a phone, andone data property to store the phone number as pictured in Figure 4.12. This class isused for adding phone devices to a person.

Figure 4.13: Camera class, with object and data properties.

The Camera class includes two objects properties to link a location with a cameraand to associate video data with a camera. It also includes two data properties storingthe cameras IP address and the path to the video source as pictured in Figure 4.13. Thisclass is used to store information about IP cameras to be use with the face recognitionapp.

The Data class separates data types from each other. It has one sub class, Video-

Data, and more sub classes can be added if more data types occure.

Figure 4.14: VideoData class, with object and data properties.

The VideoData class, pictured in Figure 4.14, includes two object properties, link-ing a camera to a specific video, and to associate people with a video. It also includestwo data properties, to a specify the videos storage path, and to flag the video. Theclass is used to store data specific for a video file stored by a camera, and to store thepeople found by the face recognition app.

The Event class separates event types from each other. It has two sub classes,GoogleCalendarEvent and FoursquareCheckin.

33

Figure 4.15: GoogleCalendarEvent class, with object and data properties.

The GoogleCalendarEvent class, pictured in Figure 4.15, includes one object prop-erty to link a Google calendar account to an event. It also includes four data propertiesto store the location, start- and end time and the title of an event. It is used to storeevents from the Google Calendar.

Figure 4.16: FoursquareCheckin class, with object and data properties.

The FoursquareCheckin class, pictured in Figure 4.16, includes one property tolink a Foursquare account to a check-in. It also includes four data properties to storethe address, city, venue and time stamp of a check-in. It is used to store a person’sFoursquare checkin.

The LocationData class separates location types from each other. It includes twosub classes, UserLocation and CameraLocation.

Figure 4.17: UserLocation class, with object and data properties.

The UserLocation class, pictured in Figure 4.17, includes one object property link-ing a person to a location. It also includes four data properties to store the address,

34

city and venue of a location as well as a timestamp of when the location was asso-ciated with a person. It is used to store a person’s locations, it can be a copy of theinformation of another location such as a foursquare check in.

Figure 4.18: CameraLocation class, with object and data properties.

The CameraLocation class, pictured in Figure 4.18, includes two object propertiesfor linking a floor and a building to a specific camera. It also includes one data propertyto store the camera venue. It is used to store camera locations for cameras in theapplication.

The Account class separates account types from each other. It includes two subclasses, FoursquareAccount and GoogleCalendarAccount.

Figure 4.19: FoursquareAccount class, with object and data properties.

The FoursquareAccount class, pictured in Figure 4.19, includes two object proper-ties linking a Foursquare checkin to a Foursquare account and for linking a person toan account. It is used to store Foursquare account information, and it does however notinclude any authentication information, as the Smart-M3 SIB database does not supplysufficient security to store such information.

Figure 4.20: GoogleCalendarAccount class, with object and data properties.

The GoogleCalendarAccount class, pictured in Figure 4.20, includes two objectproperties for linking an event to a Google Calendar calendar and for linking a personto an account. It also includes one data property storing the identification string of a

35

specific calendar. It is used to store Google Calendar account information and as withthe FoursquareAccount class, this class does not store authentication information, asthe Smart-M3 SIB data base is not secure enough.

4.5.3 Face detection and recognition

To integrate face detection and recognition, the OpenCV library was used. It providesseveral common computer vision algorithms, and has a good community support. Asthe amount of algorithms created for the purpose of detecting and recognizing faces areextensive we opted to use a ready-made implementation developed by Philipp Wagner[2]. His solution uses haar cascades for detection and local binary patterns for recogni-tion. The solution by Wagner also includes two applications alongside his library fileswith his algorithm. The applications include a full example on how to use his library,as well as a face extractor which automatically stores images of detected faces in avideo stream.

For collecting data and creating a database of faces, we installed an IP-camera inour lab corridor which sent ten second video clips every time movement was detected.The video clips where then manually checked if people of interest where in them,and then run through the face extractor application. All images were then categorizedaccording to the person in the image, and put in distinct folders with the name of theperson’s username. This way the face recognition application would be able to draw theconclusion between user and face. The extractor uses OpenCV’s CascadedDetector

class which uses haar cascades for detecting certain objects. The haar cascades areXML files with data extracted from a training set. The training is done feeding severalhundreds of images into a classifier which then creates the cascade files. Most of theimages include the object that is to be recognized, called positive examples, and someof the images with the object not present, creating negative examples. These cascadefiles can be time consuming to create, and in our case, collecting images of faces cantake a long time, and processing these images, when the cascades are created, couldtake several days. For this reason OpenCV has included multiple cascades for detectinghuman features in their library. These cascades can then be used in applications thatrequire detection of features such as the face, mouth, eyes or the upper-body of ahuman.

The face recognition application uses the images collected from the face extractor

36

Figure 4.21: Original Local Binary Pattern description [2].

to create a predictive model to recognize people. The recognition application uses Lo-cal Binary Patterns (LBP) to recognize objects, in our case faces. The original LBPalgorithm went through every pixel in an grayscale image, and looked at its neigh-boring pixels. The intensity of the pixel is then compared to its neighbors and if theneighbor has a higher intensity it gets assigned the number 1, else 0. This will create apattern of only once and zeroes, as shown in Figure 4.21 [2]. This technique howeverwas shown not to be efficient at encoding details differing in scale. This led to thedevelopment of circular LBP, which goes through all the pixels as the original imple-mentation, but instead of selecting neighboring pixels, a circle of points are createdaround the pixel. The amount of points and the radius from the center pixel can beselected freely, however the more points selected the higher computational power isneeded. If a point corresponds to a pixel, its intensity is compared to the center pixels,and as in the original algorithm, if the intensity is higher than the center pixels it isassigned a 1, else a 0 as pictured in Figure 4.22. If a point does not correspond to apixel, say it covers two, the correct pixel to compare with is chosen based upon bilinearinterpolation [2]. When the pattern is created the object is classified by comparing thehistogram of the image with histograms created from the images in the training set. Ifthe histograms match, or are a close match, and object or in our case a person has beenrecognized [3].

37

Figure 4.22: Circular Local Binary Pattern description [3].

We used IP cameras for our main video source in Krang, and all testings were adone with a NorthHQ NQ-9006 Wi-Fi enabled camera. The camera can stream bothMPEG-4 and MJPEG at the same time at a resolution of 640x480 at 30 frames persecond. To be compatible with as many cameras as possible we only used the MJPEGstream of the camera, as most IP-cameras support this format. For capturing data fromthe video stream we created a small helper library, ipcamera.py, which includes twoclasses, one to connect an IP camera, and another to handle the saving of a video clipfrom a stream. These classes in turn uses OpenCV as well the SimpleCV framework,which is built upon OpenCV. SimpleCV includes ready-made functions to connect toJPEG streams, such as the one from an IP camera supporting MJPEG. It also supportsauthentication if the video stream is password protected, a feature usually active onmost IP cameras.

We store data from the video stream as video clips, this gives us more control overthe processing of the video. It also solves the problem when more than one videostream needs to be processed, and only one computer is processing the video. We canthen queue up several video clips and process them in order without spawning morethan one video processing task, giving us a first-come-first serve processing structure.

38

Another benefit is that we can go back and forth in the video if needed, without the needto create a large buffer as it would be required if the stream was directly processed.

4.5.4 Foursquare

As described in section 3.2 and subsection 2.1.4, Foursquare is a location serviceswhere users share their locations with their friends. It also offers a push API calledthe Foursquare Real-Time API, which pushes information about user’s check-ins to aserver specified by the developer. Foursquare uses OAuth for application authentica-tion, and also forces the communication to be encrypted by not accepting any otherpush addresses other than secure HTTP ones. As we had already configured Nginx torun over SSL, no special precautions was needed to create our Django app.

Figure 4.23: Flow diagram for OAuth 2.0 [4].

For authentication we wrote an OAuth authenticator which includes the FSUser

database model and one callback function. To authenticate with our application theuser will first need to click a link including our applications client ID, it will redirect tothe Foursquare web page four user authentication, and allow our application access tothe users check-ins. Foursquare will then send the user back to an application specificURL, including an user code. Our application will then respond to the call with anapplication specific client secret and the user code, if the code and the secret are correct,Foursquare will respond with a user access token as pictured in Figure 4.23. Theaccess token is then stored in the database and can then be used to make calls to the

39

Foursquare server, and get information about the user. As the Smart-M3 SIB does notsupport authentication, and storing user sensitive data would create a security issue,the access token is not stored in the Smart-M3. The access token itself was not used inour application, but has to be fetched in order to give access to Foursquares Real-TimeAPI.

Figure 4.24: Flow diagram for Foursquare Real-Time API.

As pictured in Figure 4.24, the API works by sending out POST request with at-tached JSON data which includes information about check-in data every time an au-thenticated user checks-in. The URL Foursquare sends the request to is user specifiedwhen creating a developer account at Foursquare. One is also supplied with a pushsecret, which will be included in all push requests when creating an account. The pushsecret is used to validate that the data sent to the server is coming from Foursquare andnot a third party source.

To store users check-ins the FSHistory model is used in the relational database andthe class FoursquareCheckin in the SIB. Storing of a check-in data happens at the sametime as the Foursquare server pushes the data to the server. The function handling thepush request first checks the push secret and then checks if the user has set any of thesettings parameters described in the FSSettings. After the checks the data is stored inthe rational database as well as in the SIB. If the user has agreed to update his statusaccording to his Foursquare updates, the users current location is changed to the oneof the check-in. In the rational database as well as in the SIB this happens by settingthe users current venue to the venue of the check-in.

4.5.5 Google Calendar

Google Calendar is mostly used to store event information and meeting management.If the user has entered location information about an event we could also use that forlocating a user based on the current time and the event data. Google offers a PythonAPI for managing user events, and also offers a package for handling authentication

40

in Django. As with Foursquare, Google Calendar uses OAuth for authentication, andapplications need an access token for every user to be able to access their data. Weused the Google provided OAuth2Client package which has ready-made models tostore user credentials in Django as can be seen in Figure 4.4 in section 4.5.1.

As with Foursqurare, when a user want to add their Google Calendar account to ourapplication they first need to authenticate with Google and give us permissions to theiruser data. This is done in the same way as with Foursquare, but only using the Googleprovided OAuth2Client. The process of authenticating follows the same structure asin the Foursquare app seen in Figure 4.23. The user is sent to Google’s authenticationservers, which asks the user for credentials, where after a call is made to our applicationwith a code. The callback with a code is then sent, in addition to our applicationspecific client ID and client secret to Google’s servers which responds with the user’saccess token. There are however two main differences from the Foursquare app: A)we can pass the entire request from Googles callback to a function in OAuth2Clientpackage which handles the extraction of JSON data posted by Google and returnsthe user credentials. These credentials are then stored in a custom CredentialField

field also supplied by the OAuth2Client. B) Google grants access to multiple APIsthrough a single OAuth server, which mean the scope that we want access to needs tobe specified. One also needs to specify if the application requires offline access, whichallows the application to retrieve information even if the user is not currently usingthe application. The scopes used in our application where calendar, userinfo.profile,userinfo.email as well as offline access. These scopes give our application access tothe user’s calendars, as well as their user profile and email.

As users may have more than one calendar in their account, after authenticationwith Google we let the user decide which calendar to use and stores it in a characterfield in the database. This calendar can then be used to access upcoming events andtheir locations, as well as to be shown on their profile page.

4.5.6 Åbo Akademi Department of Information Technology

To make the application more personal, information from the web page of Departmentof Information Technologies at Åbo Akademi was integrated into Krang. The pageincludes information such as photos, e-mails, phone, and room numbers for most ofthe workers at the Department of Information Technologies. A user can also log intothe web page for additional information such as door display info.

41

Figure 4.25: Door displays in the ICT building.

The web page was to be used for its information, as well as a mean of authenticationfor the Krang application. We will now go through the two separately and explain howinformation from the page was extracted and used.

Firstly we were interested in using the page as a means of authentication to theapplication. By doing so, we eliminated the need to create a registration page forour application, and users could use their existing credentials to log in to Krang. Thealgorithm used for the authentication is shown in listing 4.1. In the application, thisalgorithm is run at the first time when the user tries to log in. If the log in is a successthe password and username is stored in the local database, else an error message isshown to the user.

1. Fetch the log in page

2. Find the log in form

2. Enter log in credentials in to the form.

3. Submit the form

4. Check if a <dl> tag with the class "portalmessage error"

exists in the returned page

5. If the element exists return False as the log in failed,

else return False

Listing 4.1: Department of Information Technology at Åbo Akademi loginalgorithm

In practice the algorithm is performed by a web scraper written in Python using

42

the third party packages Mechanize and BeautifulSoup. Web page fetching is done byMechanize and HTML parsing is handled by BeautifulSoup. This method is only runwhen a user logs in the first time. An example implementation can be seen in appendixB.1.

Secondly we wanted to extracted information from the web page. This was donein two steps, first by accessing the database storing the information shown on the webpage, and then by creating a crawler to download the person profile pictures. Theinformation for a worker stored in the database includes name, phone number, roomnumber and email as well as information about the door display status.

The door displays are custom build displays, fixed next to the doors of most roomsin the Åbo Akademi University part of the ICT building in Turku, Finland. Thesedisplays include three LEDs, one red, one yellow and one green, as well as a smalldisplay, and a button, as shown in Figure 4.25. A user can control it either by a remotecontrol in their rooms or via a web interface. One can also reconfigure the remotecontrol via the web interface, as well as change the text on the screen or add smallimages. If granted access to the database where the door display data is stored, one canchange the displays information from there by changing display specific values whichholds the current light status and the text displayed on the screen.

All but the profile picture was gathered by a Python script using the SQLAlchemyPython package for accessing the database. A wrapper for the database was also cre-ated, including functions for changing values in the database, and checking the doordisplay status.

For downloading the profile picture the standard Python package urllib2 was usedto fetch the image of a user. We found that the image name was the same as user’susername, making it trivial to figure out the URL to the image. It is then stored in theapplications media storage on the file system.

With the information gathered from the web page, and the connection to the database,we get the knowledge of where a person work desk is located, and what the personlooks like. This information can later be used in applications such as estimating wherethe person will be next and for face recognition. The door display can get informationabout a person’s whereabouts as he or she might write it on the display, and chang-ing the lights on the door signals if the person is currently located in his room or not,making it easier to track that person.

With being able to change the lights and text of the door display we can use them

43

to notify an users of a person’s current status. For instance if the person is away fromthe office, the lights can be changed to red and the text is change to "not available".

4.6 Front-end design of Krang

The front-end of Krang was developed using the Twitter Bootstrap as the main front-end framework. Their starter template was used as a base HTML template, and jQuerywas used as the main java script library. The front-end was created with usability andminimalism in mind, and was created to be used as an extension of the research.it.abo.fiwebpage, to show more information about the people working at the Department of In-formation Technologies at Åbo Akademi University. The core view of the applicationis shown in Figure 4.26. If not stated otherwise, all calls being made from the front-end to the back-end are made using Ajax. This means that the requests are done in thebackground and pages does not have to reload when a request is made. All requestsare either done by jQuerys ajax, get or post methods.

The core design of the page includes a navigation bar at the top, including the nameof the application, as well as quick links to currently available pages of the users. Tothe far right of the navigation bar is the user management, if the user is not logged init shows log in prompt, else it includes links to the users profile and settings as well asa log out link. The rest of the page contains the content of the current page viewed. Afooter including copyright information is also present on every page viewed.

The landing page, when first accessing the application, was design to get the userauthenticated with the system as fast as possible. It is also the page that non-authenticatedusers are redirected to if they try to access a page that requires authentication. It in-cludes a welcome message, and a large log-in button. The button, when pushed, opensthe log in prompt in the top right corner.

4.6.1 Profile

The profile page consists of three main parts, a quick view of the user, a user calendarand a history section as shown in Figure 4.26. The quick view includes an image of theuser, which is fetched from the research.it.abo.fi page, and some information about theuser. The information includes the name, door lights, location, status, phone number,room number and e-mail, as well as social media locations in which the user is active.The door lights are created to resemble the lights outside each person’s door, including

44

three small icons corresponding to each light on the door display. If the light outsidethe door is on, that is also shown on the webpage. If a light is not on, it is shownas grey on the users profile page. If the user is logged in these icon are interactive,and one can click on them to change their state. The location and status texts are alsointeractive if the viewer of the profile is also the owner of it. The fields are created asspan elements, with the HTML5 property contenteditable set to true. This gives themthe property of being editable if a user clicks on the text. Whenever a change is madein the quick view, a save button appears below the information. This button is onlyshown if a change is made, and if clicked hides away after some seconds.

Figure 4.26: User profile page in Krang with functional elements marked.

The calendar is created by a template tag, which fetches information from theGoogle Calendar app. This means that the calendar is defined in its own template,and then imported by the profile page. It lists the dates of the current month, withoutany weekday or month information. It was made to look as simplistic as possible,to follow the applications minimalistic design. The calendar dates color is black bydefault and light blue if an event is scheduled for a certain date. The events corre-spond to the events located in the Google Calendar that is chosen on the settings page.When clicking on a date with an event, that day’s events are listed below the calendar,including the name of the event as well as its start and end time.

45

The history section is located below the quick view and the calendar, and includesall the events and tasks that the user has been part of. This includes calendar events,foursquare check-ins or of the user has been recognized by a camera. If the user islogged in, and hovers over an entry in the history a cross icon appears which lets theuser delete its history if need be.

For creating a more complete look, some buttons and text areas were inserted inthe profile page. These includes a text area containing information about what theuser is doing at this moment, giving a look at what the system could be able to do byadding more information sources and also a section for sending messages to anotheruser and a button to follow a user’s updates. These are only esthetic features, and haveno function, the front-end design is however ready if these functionalities were to beadded to the back-end at a later time.

4.6.2 Settings

The settings page includes two sections, one to add social media information, and oneto link other services to Krang as shown in Figure 4.27. The social media section in-cludes several well-known social media services such as Facebook, Linked-In, Twitterand Google+. The user can insert their username or an URL to their profiles, whichwill then show up on their profile page. If a username is inserted, the page tries torecreate the link to the user profile on that particular service. If this fails, the user canmanually enter the URL. To save the information a save button is located beneath thesection.

46

Figure 4.27: User settings page in Krang.

The account linking section consists of three buttons, one for each service currentlyimplemented, these being Foursquare, Google Calendar and face recognition. Whileface recognition is part of Krang itself, it is created as an opt-in feature due to moraland ethical reasons. When clicking either the Foursquare or Google Calendar buttonthe user will be sent to the services authentication page, where they will be promptedto insert their username and password. If the users credentials are correct they willbe sent back to the settings page, and the button will change color from blue to greensignaling that the accounts has now been linked. Clicking the face recognition buttonwill instantly turn the feature on, and the user will be tracked. After enabling any of theservices an unlink, and settings button will also appear to the right of the link button.

The unlink button will simply unlink the service with Krang, and remove any in-formation connected to it. The settings button will show a settings modal window foreach individual service. For Foursquare the users have the choice of Krang not updat-ing their status when a check-in is made, or to only update the status if the location isrelated to their work, in this case the ICT building in Turku. For Google Calendar theuser can select which calendar to use, something which is essential for the system towork. Face recognition settings include the ability to temporarily disable tracking, orto only update the status but not add it to the history.

47

4.7 Use Cases

This section covers use-cases of the Krang demonstrator, and shows how it can be usedto track people. By combining technologies, we can create uses that otherwise wouldbe hard to implement. All cases include a flow chart to represent the flow of eachuse case, as well as the service or application used in each step. The flow charts alsoinclude a legend describing the different icons used in the chart.

4.7.1 Complementing location information

Figure 4.28: Flowchart of complementing location information use case.

As our system integrates several ways of tracking people, the information from a per-son’s new location might include information from the current location. If this is thecase the two locations can complement each other to create more information about alocation as shown in Figure 4.28. This way the system knows more about a locationthan it did with only one source of information.

If the application gets new information about a user’s whereabouts it will first com-pare the new location to the old one. If the location address or venue is the same, theapplication checks if the new location includes additional information to the current

48

one. If this is the case the additional information is added to the user’s current userlocation, if not the new location is discarded and the old one is kept. If however thenew location does not match either the address or the venue of the old one, then theuser’s location is updated to the new location.

This kind of behavior can be useful if users use several different ways to broadcasttheir location, for example Foursquare, GPS and Facebook. If a camera has recognizedthe person, the location of the camera can also be added to the user current location,as additional data. This way the application can get an as accurate location as possibleof a user, including not only the venue or address but also the exact floor, room andcoordinates. Applications benefiting from this kind of information can be a patienttracking system, when patient with dementia or Alzheimer’s could be tracked in a hos-pital. Other more general uses can be exact package tracking, where the receiver wouldknow exactly where their package or for tracking workers, to know where employeesare currently located.

49

4.7.2 Event notification system

Figure 4.29: Flowchart of event notification Celery task.

Django comes with a build in SMTP client which, if configured, can send out mails to asingle or multiple addresses. Krang uses the mail system for sending out mails if a userwas going to be late to a meeting. This was done by creating a celery task, as picturedin Figure 4.29, that at a regular schedule checks if there is an upcomming event for allusers authenticated with Krang. If an event was scheduled for the next 15 mintutes,Krang checks if the user has a current location. If that is the case, a comparison oflocations is made, and if they are not the same, a calculation of the distance and drivetime from the users current location to the one of the events is made. This is doneusing the Google Distance Matrix (GDM), which takes two addresses returning thedriving distance, as well as the driving time for that route. A calculation if the userscould make it in time if he was currently driving is then made, by taking the delta ofthe current time and the driving time added the to time when the users last location was

50

set, as shown in algorithm 1.If the time delta is negative, the users is going to be late to the meeting. Krang will

then check if there are other participants who has responded to the event. If this is thecase, these peoples email addresses are gathered in a list and a mail will be sent out tothem contaning information about who is going to be late, as well as by how much.

This kind of systems can be useful for companies or organization to message meet-ing or event participants about people who are going to be late. If deployed at a largescale it could also be used for workers who have set appointments with clients, to no-tify if they are going to be late. An example use case for this would be a hostipal wheredoctors could be notified if patients are about to be late, and let the doctor take anotherclient in between.

Algorithm 1 (Late time calculation)

∆T = ct− (dt + ut)

51

4.7.3 Information display with user data

Figure 4.30: Flowchart of a Celery task to change a status display using multipledata sources.

Information displays or lights, such as the one described in subsection 4.5.6, whichlets people display their current status or whereabout can be installed outside officesor meeting rooms to let others know if the person is busy or if the room is currentlyoccupied. These status displays often have to be manually changed to the right status,and are often left unused, as users forget or simple does not bother changing the status.Our system can with the help of people tracking and face recognition set the statusof such displays without user interaction. Figure 4.30 shows a flowchart how such assystem works.

In the chart the application first check for events in a meeting room, given a interval.

52

By creating an Celery task that runs at a regular schedule. The task would then checkuser calendars, and looking for events with the same location as the room. If suchevents are found then check if the current time is between the start and end time ofthe event. If this is the case the information display changes to occupied by setting acertain led or showing text on a display. The application could also be configured tocheck for an upcoming event and change the status in before it occurs.

If however there are no events schedules for a room, an alternative route could betaken, the application could check for people inside the room instead. This can be doneby looking for faces inside the room. This can be done by first checking for camerasinside the meeting room, and then starting to stream data from the camera back to ourapplication which processes the video. If a face, or several faces are found in the room,then it might be in use. To be sure, a re-check timer can be set to check the status ofthe room some minutes later. If there are still faces found in the room, it is probably inuse, and we can se the information display to occupied. This means that the applicationcould work even if the people inside the room are not users of the application, and wedon’t have access to their calendars.

Systems like this could be used in offices buildings where big companies, or whereseveral companies share the same spaces. The application could also be linked toan existing booking server, which would show if a room is currently in use, or if abooking can be made immediately. Other uses could also be in aiding larger schoolswhere lecturers need to book rooms in advance, by showing rooms currently in use andhistory about how rooms have been used in the past.

4.8 Web SIB Explorer and Smart-M3 Tutorial

Developers using Smart-M3 for the first time might find the learning curve a bit toosteep as there is almost no documentation to read, and only a few open-source projectsuse the Smart-M3. For this reason we created a small tutorial introducing new usersto Smart-M3 in parallel with creating Krang. It goes through the process of creating asmall home automation application using the Python kernel processing interface (KPI).The tutorial was releases as a TUCS technical report, and can be accessed at WHEREIS THE TUTORIAL ?!?!

53

Figure 4.31: Web SIB Explorer Class Tree view.

We also noticed that when creating and working with ontologies for the Smart-M3,debugging became a time consuming task. One would have to write code for testingqueries, and create parsers for the results, in order to debug. To solve this problem theWeb SIB Explorer was created, which enabled us to get a quick overview of the SIBand to test out queries and subscriptions with ease. It was created as a web application,made to run locally, using the small Python web framework Flask.

The application includes features such as:

1. Listing all classes and properties.

2. A class tree view for navigating ontology classes and their objects, as well asdelete objects as can shown in Figure 4.31.

3. A querying feature using the CodeMirror code editor for syntax highlighting andbracket matching functionality.

4. Subscription handling using a SQLite database for storing current subscriptionsand data about them.

5. A small benchmark feature to give a quick overview of how fast the SIB wasrunning on different platforms.

The application can be accessed at the applications github repository at http://eslab.github.com/Web-SIB-Explorer/.

54

5 DISCUSSION

5.1 Conclusions

By using multiple sources of data we created a web service which was able to trackpeople, given a user’s consent. The program integrates technologies such as Foursquare,Google Calendar and face recognition for getting the user location. All technologiesintegrated into Krang existed before the creation of our solution. We created Krang asa glue between the services and devices, to make them aware of each other. In total weused over ten existing technologies for creating one unified smart system.

For this we had to use storage solutions that integrate well with software usingseveral data sources. The program uses a dual database setup, using a PostgreSQLdatabase for redundancy as well as for storing sensitive data such as user credentialsand a Smart-M3 triple store to store all non-sensitive information. By using Smart-M3ssubscribe feature, other devices could also act upon changes in the database, whichmakes our software more modular. For storing image and video data we opted to usethe file system of the operating systems to store the actual data and store Meta-dataabout for files in the databases. This gives more control over the data, and if requiredenables easy exportation of media files.

We demonstrated how other devices and systems can use the data collected by oursoftware for their benefit. Status displays outside offices could be change according tothe user’s current location, and other users could be notified if someone is going to belate to an event or meeting. This required integrating several other technologies such asGoogle Distance Matrix, and SMTP. It also showed how existing systems could benefitfrom ours.

Firstly we have shown that by combining technologies and by making them awareof each other we could create a more intelligent system. Secondly, new solutions tocreate data for smart-spaces might be redundant, as usable source of information mightalready exist, but used in a different way. Companies such as Facebook have stored

55

user data and their activities for several years, and are now starting to show the data’strue possibilities. Facebook Graph Search is a feature in which users can use naturallanguage to search for information about other people on Facebook. The feature isusing the vast amount of information gathered from its users; data that they had accessto several years ago, but have only now been using. With the increasing interest indata mining, the internet of things as well as combining data from multiple sources,computers might soon know more about our lives than we do.

5.2 Future work

While adding more data sources is always an improvement to a tracking system, amore important feature would be the ability to create rules. Features presented in thethesis are all hard coded examples, and there is no syntax for creating rules. Creatingrule syntax, and an interpreter for this, would greatly improve the usability of theapplication. To let users create their own rules would also add to the usability, as wellas user-friendliness of the application. A suggested syntax for the rules would be tohave an if-this-then-that structure, where the user would first select a trigger, this, andthen add an action, that, if the trigger gets a hit.

Other improvements include probabilistic models, in which the system could pre-dict future movement of its users. This would open up possibilities such as reschedul-ing of events if the system notices an abnormality in the predicted route a user hastaken, or to create a more effective daily work routine for people on the run. Thiscould also be applied in other areas such as tracking packages or as an early warningsystem for planes and boats.

56

BIBLIOGRAPHY

[1] Bitmanufaktur GmbH. Openbeacon active rfid project. URL: http://www.openbeacon.org/OpenBeacon_Active_RFID_Project.

[2] Philipp Wagner. local binary patterns. URL: http://www.bytefish.de/blog/local_binary_patterns.

[3] Ville Arola, Niko Kurvinen, and Eero Siniluhta. Local binary pattern. URL: https://noppa.lut.fi/noppa/opintojakso/ct50a6100/lisatty/local_binary_pattern.pdf.

[4] Social Lipstick. Under the covers of oauth 2.0 at facebook, 5 2010. URL: http://www.sociallipstick.com/?p=239.

[5] Peter Murray. Heathrow airport to adopt face scanners toscreen passengers. URL: http://singularityhub.com/2011/08/06/heathrow-airport-to-adopt-face-scanners-to-screen-passengers/.

[6] Gerrit Hornung. The european regulation on biometric passports: Legislative pro-cedures, political interactions, legal framework and technical safeguards. URL:http://www.law.ed.ac.uk/ahrc/script-ed/vol4-3/hornung.asp.

[7] A.C. Weaver. Biometric authentication. Computer, 39(2):96 – 97, feb. 2006.doi:10.1109/MC.2006.47.

[8] S. Liu and M. Silverman. A practical guide to biometric security technology. ITProfessional, 3(1):27 –32, jan/feb 2001. doi:10.1109/6294.899930.

[9] T. Ahonen, A. Hadid, and M. Pietikainen. Face description with local binarypatterns: Application to face recognition. Pattern Analysis and Machine Intelli-gence, IEEE Transactions on, 28(12):2037 –2041, dec. 2006. doi:10.1109/TPAMI.2006.244.

[10] L. Wiskott, J.-M. Fellous, N. Kuiger, and C. von der Malsburg. Face recog-nition by elastic bunch graph matching. Pattern Analysis and Machine Intelli-gence, IEEE Transactions on, 19(7):775 –779, jul 1997. doi:10.1109/34.598235.

57

[11] M.A. Turk and A.P. Pentland. Face recognition using eigenfaces. In Computer Vi-sion and Pattern Recognition, 1991. Proceedings CVPR ’91., IEEE Computer So-ciety Conference on, pages 586 –591, jun 1991. doi:10.1109/CVPR.1991.139758.

[12] Andrew J. Hopwood, Cedric Hurth, Jianing Yang, Zhi Cai, Nina Moran, John G.Lee-Edghill, Alan Nordquist, Ralf Lenigk, Matthew D. Estes, John P. Haley,Colin R. McAlister, Xiaojia Chen, Carla Brooks, Stan Smith, Keith Elliott, PierisKoumi, Frederic Zenhausern, and Gillian Tully. Integrated microfluidic systemfor rapid forensic dna analysis: Sample collection to dna profile. AnalyticalChemistry, 82(16):6991–6999, 2010. URL: http://pubs.acs.org/doi/abs/10.1021/ac101355r, doi:10.1021/ac101355r.

[13] Fan Jing and Jiao Ruili. New fast algorithm on dna sequence alignment. InBioinformatics and Biomedical Engineering , 2009. ICBBE 2009. 3rd Interna-tional Conference on, pages 1 –4, june 2009. doi:10.1109/ICBBE.2009.5163102.

[14] L.P. Hewlett-Packard Development Company. Hp notebook pcs - using finger-print reader to login to applications and web sites. URL: http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c00958213.

[15] Ms. Smith. Laptop fingerprint reader destroys ’entire security model ofwindows accounts’. URL: http://www.networkworld.com/community/blog/laptop-fingerprint-reader-destroys-entire-security-model-windows-accounts.

[16] Mislav Grgic and Kresimir Delac. Face recognition homepage: Algorithms.URL: http://www.face-rec.org/algorithms/.

[17] D. Ramanan, D.A. Forsyth, and A. Zisserman. Strike a pose: tracking people byfinding stylized poses. In Computer Vision and Pattern Recognition, 2005. CVPR2005. IEEE Computer Society Conference on, volume 1, pages 271 – 278 vol. 1,june 2005. doi:10.1109/CVPR.2005.335.

[18] D.M Gavrila. The visual analysis of human movement: A survey. ComputerVision and Image Understanding, 73(1):82 – 98, 1999. doi:10.1006/cviu.1998.0716.

[19] T.L. Berg, A.C. Berg, J. Edwards, M. Maire, R. White, Yee-Whye Teh,E. Learned-Miller, and D.A. Forsyth. Names and faces in the news. In Com-puter Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004IEEE Computer Society Conference on, volume 2, pages II–848 – II–854 Vol.2,june-2 july 2004. doi:10.1109/CVPR.2004.1315253.

[20] Microsoft Corporation. Kinect for windows. URL: http://www.microsoft.com/en-us/kinectforwindows/.

58

[21] Michael Zollhöfer, Michael Martinek, Günther Greiner, Marc Stamminger,and Jochen Susmuth. Automatic reconstruction of personalized avatars from 3dface scans. Computer Animation and Virtual Worlds, 22(2-3):195–202, 2011.URL: http://dx.doi.org/10.1002/cav.405, doi:10.1002/cav.405.

[22] T. Leyvand, C. Meekhof, Yi-Chen Wei, Jian Sun, and Baining Guo. Kinectidentity: Technology and experience. Computer, 44(4):94 –96, april 2011.doi:10.1109/MC.2011.114.

[23] K.K. Biswas and S.K. Basu. Gesture recognition using microsoft kinect. InAutomation, Robotics and Applications (ICARA), 2011 5th International Con-ference on, pages 100 –103, dec. 2011. doi:10.1109/ICARA.2011.6144864.

[24] Bangpeng Yao and Li Fei-fei. Grouplet: A structured image representationfor recognizing human and object interactions. In Computer Vision and Pat-tern Recognition (CVPR), 2010 IEEE Conference on, pages 9 –16, june 2010.doi:10.1109/CVPR.2010.5540234.

[25] M. Van den Bergh, D. Carton, R. de Nijs, N. Mitsou, C. Landsiedel, K. Kuehn-lenz, D. Wollherr, L. van Gool, and M. Buss. Real-time 3d hand gesture interac-tion with a robot for understanding directions from humans. In RO-MAN, 2011IEEE, pages 357 –362, 31 2011-aug. 3 2011. doi:10.1109/ROMAN.2011.6005195.

[26] Simon Hay and Robert Harle. Bluetooth tracking without discoverabil-ity. In Tanzeem Choudhury, Aaron Quigley, Thomas Strang, and KojiSuginuma, editors, Location and Context Awareness, volume 5561 of Lec-ture Notes in Computer Science, pages 120–137. Springer Berlin Heidelberg,2009. URL: http://dx.doi.org/10.1007/978-3-642-01721-6_8, doi:10.1007/978-3-642-01721-6_8.

[27] Ian Paul. Google street view’s wi-fi snooping engineer is outed. URL:http://www.pcworld.com/article/254774/google_street_views_wi_fi_snooping_engineer_is_outed.html.

[28] T. Mori, Y. Suemasu, H. Noguchi, and T. Sato. Multiple people tracking byintegrating distributed floor pressure sensors and rfid system. In Systems, Manand Cybernetics, 2004 IEEE International Conference on, volume 6, pages 5271– 5278 vol.6, oct. 2004. doi:10.1109/ICSMC.2004.1401031.

[29] Frank Miskelly. Electronic tracking of patients with dementia and wan-dering using mobile phone technology. Age and Ageing, 34(5):497–499,September 2005. URL: http://ageing.oxfordjournals.org/content/34/5/497.short,arXiv:http://ageing.oxfordjournals.org/content/34/5/497.full.pdf+html, doi:10.1093/ageing/afi145.

59

[30] K. Michael, A. McNamee, and M. Michael. The emerging ethics of humancentricgps tracking and monitoring. In Mobile Business, 2006. ICMB ’06. InternationalConference on, page 34, june 2006. doi:10.1109/ICMB.2006.43.

[31] Ingrid Burbey. Predicting future locations and arrival times of individuals.

[32] SARA MALM and LUCY OSBORNE. Mobile phone companiescan predict future movements of users by building a profile of theirlifestyle. URL: http://www.dailymail.co.uk/sciencetech/article-2190531/Mobile-phone-companies-predict-future-movements-users-building-profile-lifestyle.html.

[33] Will Oremus. What happens when our cellphones can predict our every move?URL: http://www.slate.com/blogs/future_tense/2012/08/08/cellphone_tracking_what_happens_when_our_smartphones_can_predict_our_every_move_.html.

[34] D. Ashbrook and T. Starner. Learning significant locations and predicting usermovement with gps. In Wearable Computers, 2002. (ISWC 2002). Proceedings.Sixth International Symposium on, pages 101 – 108, 2002. doi:10.1109/ISWC.2002.1167224.

[35] M. Bassick, T. McNamara, and D. Sullivan. Employee surveillance: An ethicalconsideration. The Ethical Imperative in the Context of Evolving Technologies,page 77.

[36] Django Software Foundation. The web framework for perfectionists with dead-lines. URL: https://www.djangoproject.com/.

[37] Ask solem and contributors. Celery: Distributed task queue. URL: http://www.celeryproject.org/.

[38] Inc. VMware. Rabbitmq - messaging that just works. URL: http://www.rabbitmq.com/.

[39] NETCRAFT LTD. January 2013 web server survey. URL: http://news.netcraft.com/archives/2013/01/07/january-2013-web-server-survey-2.html.

[40] Mondotech. Apache 2.4 vs nginx benchmark showdown, February. URL: http://mondotech.blogspot.fi/2012/02/apache-24-vs-nginx-benchmark-showdown.html.

[41] S.Dmitrij. Nginx vs cherokee vs apache vs lighttpd, April 2012. URL: http://www.whisperdale.net/11-nginx-vs-cherokee-vs-apache-vs-lighttpd.html.

[42] Joshua Zhu. Apache 2.4 faster than nginx?, February 2012. URL: http://blog.zhuzhaoyuan.com/2012/02/apache-24-faster-than-nginx/.

60

[43] Unbit. uwsgi. URL: http://projects.unbit.it/uwsgi/.

[44] Benoit Chesneau. Gunicorn - python wsgi http server. URL: http://gunicorn.org/.

[45] Paul J. Davis. Why do i need nginx and something like guni-corn?, November 2011. URL: http://serverfault.com/questions/331256/why-do-i-need-nginx-and-something-like-gunicorn.

[46] nosql database.org. Nosql databases. URL: http://nosql-database.org/.

[47] The PostgreSQL Global Development Group. Postgresql. URL: http://www.postgresql.org/.

[48] Michael Larabel. Linux 3.7 file-system benchmarks: Ext4, btrfs, xfs, November2012. URL: http://www.phoronix.com/scan.php?page=article&item=linux_37_fsthree&num=1.

[49] Woodrow W. Bledsoe. The model method in facial recognition.

[50] Woodrow W. Bledsoe. A man-machine facial recognition system-some prelimi-nary results.

[51] Phillip Ian Wilson and John Fernandez. Facial feature detection using haarclassifiers. J. Comput. Sci. Coll., 21(4):127–133, April 2006. URL: http://dl.acm.org/citation.cfm?id=1127389.1127416.

[52] M.A. Turk and A.P. Pentland. Face recognition using eigenfaces. In Computer Vi-sion and Pattern Recognition, 1991. Proceedings CVPR ’91., IEEE Computer So-ciety Conference on, pages 586 –591, jun 1991. doi:10.1109/CVPR.1991.139758.

[53] P.N. Belhumeur, J.P. Hespanha, and D.J. Kriegman. Eigenfaces vs. fisherfaces:recognition using class specific linear projection. Pattern Analysis and MachineIntelligence, IEEE Transactions on, 19(7):711 –720, jul 1997. doi:10.1109/34.598228.

[54] W3Techs. Usage of javascript libraries for websites. URL: http://w3techs.com/technologies/overview/javascript_library/all.

61

SAMMANFATTNING

SUPERB RUBRIK

This chapter is left empty until the structure and text of the English version is close tocomplete.

Introduktion

Implementering

Sammanfattning

62

Appendices

63

A CONFIGURATIONS

A.1 uWSGI

1 [uwsgi]

2 socket = /tmp/<PROJECT_NAME>.sock

3 chdir = /PATH/TO/DJANGO/PROJECT

4 pythonpath = /PATH/TO/VIRTUAL_ENV

5 env = DJANGO_SETTINGS_MODULE=<PROJECT_NAME>.settings

6 module = django.core.handlers.wsgi:WSGIHandler()

7 aster = true

8 processes = 4

9 daemonize = /PATH/TO/DEAMONIZE/LOG.log

10 pidfile = /PATH/TO/PIDFILE.pid

A.2 Nginx

1 server {

2 listen 443;

3 server_name localhost;

4 ssl on;

5 ssl_certificate /PATH/TO/server.crt;

6 ssl_certificate_key /PATH/TO/server.key;

7

8

9 location / {

10 include uwsgi_params;

11 uwsgi_pass unix:///tmp/krang.sock;

12 }

13

64

14 location /media {

15 root /DJANGO/PROJECT/MEDIA_DIR;

16 }

17

18 location /static {

19 root /DJANGO/PROJECT/STATIC_DIR;

20 }

21 }

65

B WEB SCRAPING

B.1 research.it.abo.fi login

1 def itLogin(self, username, password):

2 """ Logs in to research.it.abo.fi page

3 Args:

4 username (str): Users username

5 password (str): Users password

6 Returns:

7 True if the user was successfully logged in else

False.

8 """

9

10 logger.info("Logging in with: "+username)

11

12

13 self.getPage(self.URL_LOGIN)

14 self.br.select_form(nr=1) # 2nd form = login

15 self.br['__ac_name'] = username # Set username

16 self.br['__ac_password'] = password # Set password

17 self.br.submit() # Submit form

18

19 soup = BeautifulSoup(self.br.response())

20 logger.info("Got soup")

21

22 try :

23 # Element not be present if we have logged in

24 error = soup.find('dl', {'class' : 'portalMessage

error'})

66

25

26 if not error:

27 self.logins['it'] = True

28 logger.info("All ok, returning True")

29 return True

30 else:

31 logger.info("Got an error logging in,

returning False")

32 return False

33

34 except AttributeError:

35 logger.info("Attribute error!")

36 return False

67