DEPARTMENT OF COMPUTER SCIENCE

A «CineClash»: Using Mobile Games for Building User Movie Interest Profiles

and Recommendations

Author: Seidalin Sultan

Supervisor: Dr. Lilian Blot 6th September 2016

Number of word = 19464 as counted by Microsoft Word, excluding the references and appendices

This thesis submitted for the degree of MSc in Computing

This thesis is dedicated to the memories of my

Mother Sarieva Saule (1960-2016)

You will always be in my heart, mom

Abstract

The main purpose of the project is to identify whether mobile games can be used to create user’s

movie interest profile based on Android OS. Additionally, recommender system is to be

implemented which is going to use the generated profile in order to provide the user with items

he/she could potentially be interested in. A mobile quiz game was designed and implemented

allowing: players to answer movie based questions proposed by the system, users to create their

own set of question to challenge another player, answer the questions that are sent by other gamers

and check whether suggested movies meet his/her interests. Each user’s answer or created question

is analysed and used to generate an interest profile of a player using hybrid user profiling approach,

where keyword tagging technique represents the implicit part and user feedback model represents

the explicit part of it. A recommender system is created which is based upon content based and

collaborative filtering approaches. This recommender system is then used to propose proper

movies or TV shows to users based on the user profiles created before.

Our user profiling and recommender systems are evaluated using two methods, which are:

qualitative and quantitative. Our games are asked to take a survey, which helps us to do qualitative

evaluation. Quantitative evaluation is made by using special metrics, such as Precision, Recall and

F-Measure. The results of our research are provided at the end of the project.

Acknowledgments

A special thanks to my supervisor Dr. Lilian Blot for his patience, guidance, support and for

encouraging me throughout the entire process.

I would also give my gratitude to my family for their understanding, support and faith in me

throughout this year. I would not have the opportunity to do this master course without their

support.

Additionally, I would like to thank my friend Kamashkin Sergei for helping me out with the GUI

of the “CineClash” application, and my friend Alexander Popov for helping me with creating

movie based questions.

Ethics statement

All the information gathered by the game implemented in the project is anonymous and will not

be used elsewhere under any circumstances. All the data will be encrypted and stored in a secure

place. Only student and supervisor have an access to the data gathered during the research.

Questionnaire was used to collect data from uses which is provided in Appendix A. The

questionnaire is completely anonymous. All the participants are informed that their participation

is voluntary and they can quit the questionnaire at any time. This project completed without any

ethical implication.

Table of Contents

Abstract ................................................................................................................................ 3

Acknowledgments ................................................................................................................. 4

Ethics statement.................................................................................................................... 5

List of Figures ........................................................................................................................ 8

List of Tables ......................................................................................................................... 9

List of Listings ...................................................................................................................... 10

Chapter 1. Introduction ....................................................................................................... 11 1.1. Aim ....................................................................................................................................... 12

Chapter 2. Literature Review ............................................................................................... 14 2.1. User profiling ........................................................................................................................ 14

2.1.1. User profile contents ............................................................................................................... 14 2.1.2. Obtaining user profiles ............................................................................................................ 15 2.1.3. User profiling techniques ........................................................................................................ 16

2.2. Recommender systems ......................................................................................................... 18 2.2.1. Overview .................................................................................................................................. 18 2.2.2. Recommendation System Implementation Techniques ......................................................... 19

2.3. Gamification ......................................................................................................................... 20 2.4. Current market overview ...................................................................................................... 21

Chapter 3. Requirement and Problem Analysis .................................................................... 23 3.1. Stakeholders ......................................................................................................................... 23 3.2. System requirements ............................................................................................................ 23

3.2.1. Functional Requirements ........................................................................................................ 24 3.2.2. Non-Functional Requirements ................................................................................................ 24

3.3. Database Requirements ........................................................................................................ 26 3.3.1. Movie Database Requirements ............................................................................................... 26 3.3.2. Local Database Requirements ................................................................................................. 26

Chapter 4: Design and Implementation ................................................................................ 28 4.1. System .................................................................................................................................. 28 4.2. The Client ............................................................................................................................. 30 4.3. Movie Database Design ......................................................................................................... 31 4.4. Web Server ........................................................................................................................... 33 4.5. Interaction Design ................................................................................................................. 34 4.6. User Profiling ........................................................................................................................ 35 4.7. Recommender System .......................................................................................................... 38 4.8. Security ................................................................................................................................ 41

Chapter 5. Evaluation and Testing ........................................................................................ 42 5.1. Functional Requirements ...................................................................................................... 42 5.2. Non-Functional Requirements ............................................................................................... 43 5.3. Expert Evaluation .................................................................................................................. 44

5.3.1. Cognitive Walkthrough Evaluation .......................................................................................... 44 5.4. Recommender System Evaluation.......................................................................................... 45

5.4.1. Qualitative Evaluation ............................................................................................................. 45 5.4.2. Quantitative Evaluation ........................................................................................................... 47

5.5. Summary .............................................................................................................................. 49 5.5.1. Qualitative Evaluation Summary ............................................................................................. 49

5.5.2. Quantitative Evaluation Summary........................................................................................... 50

Chapter 6. Conclusion .......................................................................................................... 51 6.1. Project Contribution .............................................................................................................. 51 6.2. Work summary ..................................................................................................................... 51

Chapter 7. Future Work ....................................................................................................... 52 Improvements in the mobile application ........................................................................................... 52 Improvements in the user profiling ................................................................................................... 52 Improvements in the recommender system ...................................................................................... 52

References .......................................................................................................................... 53

Appendix A.......................................................................................................................... 57

Appendix B .......................................................................................................................... 59

Appendix C .......................................................................................................................... 61

Appendix D ......................................................................................................................... 63

List of Figures

Figure 1.1. Share of time spent per day with major media by US adults (in %). Source: [12] .....12 Figure 2.1. Association rules discovered which describes customer’s behaviour. Source: [9] .....16 Figure 2.2. Structure of an Artificial Neural Network. Source: [15]. ............................................18 Figure 2.3. Graphical user interface of question page. From left to right: «Quizclash», Source:

[24]; «Trivia Crack», Source: [25]; «QuizUp», Source: [26]................................................22 Figure 4.1. Worldwide Smartphone OS Market Share from 2012 Q2 to 2015 Q2. Source: [29]. 28 Figure 4.2. Current and predicted market share of worldwide smartphone shipments by operating

system from 2014 to 2020. Source: [30] ...............................................................................29 Figure 4.3. Operating system versions running on devices as of August 2016. Left – Android,

Source: [31]. Right – iOS, Source: [32]. ...............................................................................30 Figure 4.4. General overview of the system. Data flow between client, server and movie

database. ................................................................................................................................30 Figure 4.5. Screenshots of mobile application created during the project development. From left

to right: Main Screen; “Start Game” screen; Question screen ..............................................31 Figure 4.6. Entity-relationship diagram of the movie database based upon our knowledge about

their database and API documentation ..................................................................................32 Figure 4.7. Model of the assumption to be made in the user profile process ................................35 Figure 4.8. Entity-relationship diagram of the database stored on our server ...............................37 Figure 4.9. Extract from the database from our server. Top – User’s preferences; Left – User’s

director profile; Right – User’s genre profile ........................................................................37

List of Tables

Table 2.1. List of tag ranked by their number of occurrence. Source: [14]. .................................17 Table 3.1.Functional requirements of the system to be implemented ...........................................24 Table 3.2. Non-Functional Requirements of the system ...............................................................26 Table 3.3. Movie Database Requirements of our system ..............................................................26 Table 3.4. Local Database Requirements of our system ...............................................................27 Table 4.1. Schema of questions table with keywords....................................................................35 Table 4.2. User preference table schema with an example ...........................................................36 Table 4.3. User's genre profile with corresponding weights .........................................................36 Table 4.4. User's director profile with corresponding weights ......................................................36 Table 4.5. User’s genre profile (Normalised) ................................................................................39 Table 4.6. User's director profile (Normalised) .............................................................................39 Table 5.1. Test cases of Functional Requirements ........................................................................43 Table 5.2. Test case results of Functional Requirements ..............................................................43 Table 5.3. Steps followed during Cognitive Walkthrough Evaluation..........................................45

List of Listings

Listing 4.1. Example of Activity class. Source: [34].....................................................................31 Listing 4.2. Extract of Fragment class which is responsible for login process .............................32 Listing 4.3. Extract from AsyncTask class which is responsible for connecting and retrieving

data from the movie database. ...............................................................................................33 Listing 4.4. Extract of PHP code located on the server which is responsible for the login process

...............................................................................................................................................34 Listing 4.5. Part of the PHP code responsible for normalisation. .................................................41

Chapter 1. Introduction

Personalisation became extremely important in Computer Science field, especially, in the

applications of Recommender Systems. Recommender Systems evolved to a very high level with

the personalisation of information. These systems are able to generate recommendations

accurately and efficiently for a user. Recommender system deals with many users where each of

them has his/her own preferences and requirements. Its aim is to meet the requirements of an

individual by recommending specific item to him/her or modifying itself based on the user’s needs.

Recommender system can benefit from user profiling by understanding the needs and

requirements of the person and behave accordingly.

User profile is a detailed description of someone’s background, goals and preferences when they

are using software application. User profile could be very different according to the context. If we

consider newspaper application, the user profile might contain topics which user prefers to read,

reading habits and patterns, etc. From one domain to another, not only user profile could vary, but

also the techniques of acquiring it might be different.

According to [1], user profiling is an acquisition of important user information that is essential for

the context that is considered. Stuart et al. [2] defined user profiling as discovering useful patterns

of user behaviour using machine-learning techniques. User profiling can be described as the

method of obtaining information about a user’s interest domain. Gathered information can be used

by the system to gain knowledge about the person which then can be used again to enhance the

retrieval even more for providing satisfaction to the user. User profiling, has two crucial phases,

which are efficiently understanding user, and recommending items of user’s interest based on the

knowledge gained. We have already discussed the first phase, however, the second one

(recommending items) is as vital as the first. Silvia et al. [1] described recommender system as a

method of recommending potentially interesting items to sole user and to the group of users with

similar tastes. Different user profiling techniques have evolved over the last years and much

research has been accomplished in the field of user profiling and recommender systems.

Significant research has been done in the area of user profiling and recommender system

implementation. Some user profiling technics and recommender systems use web data to

customise websites according to users’ needs [3], while others analyse different users’ behaviour

in social networks in order to build user profiles [4]. However, the main goal of our project is to

evaluate how mobile games can be used in this context. User interest profile will be built based on

movies and recommender system will be developed which will use the profile to recommend

movies to the potential player using patterns and principles of mobile interface design.

The emergence of wireless and mobile devices gave an opportunity for the recommender systems

and user profiling technics make a new step in evolution. One of the main advantages is that mobile

devices, especially mobile games, are widely spread worldwide and this trend continues to grow.

It should be taken into consideration that a mobile phone is a device which is personal, meaning

that whatever application installed or game played potentially contains personal information about

interests of the user. Thus, giving a huge opportunity to develop mobile games which can build a

user interest profile that can be easily used by recommender systems to improve item

recommendations.

Using mobile games as context in our research could be very useful. According to Deloitte, mobile

gaming is among the highest growing markets. Their statistics show that just in the beginning of

2016 there were 800,000 mobile games released comparing to 17,000 titles available for consoles

or PCs. Figure 1.1 clearly shows an increasing trend in mobile devices since 2012. Mobile devices

have become so popular, that classical online media usage is going down up to 2014 and this

downward trend is continuing onwards. Looking at titles in Google Play Store, which is official

market of applications on Android OS, it can be seen that mobile games are an applicable basis

for the research. For example, “Quizclash” by FEO Media AB, which is a quiz game where users

can play with each other by answering questions provided by the system on diverse topics,

including movies, has 10 million downloads. “Trivia Crack” by Etermax, which is also a question

based trivia game has 100 million downloads. Both of this examples convince that mobile games

are an appropriate foundation for research, as it is noticeable that mobile games can gather huge

audience.

1.1. Aim

The aim of this project is to study how mobile games can be used to generate user profiles and

how recommender system can benefit from user profile, to provide tactics which solve common

mobile development problems and to develop a mobile game for the Android platform that will

generate user profile oriented on movies and to implement recommender system which will use

these profiles to recommend appropriate movies to the user. The main research question that will

be answered by the thesis is:

Can we use mobile games in order extract necessary information to create user profiles and then

recommend appropriate items to the user based on the user profile?

Our first priority is to build user profiling system, which should be able to extract important

information to build a user profile of every user. Every answered or created question should be

analysed and appropriate data is to be collected. A remote database containing the generated

information is to be populated by extracted anonymous data.

Next mission is to create a recommender system which will use above mentioned user profile data

on a remote database and suggest items to the players. Recommendations will be evaluated by

giving the players suggestions and asking them if they like it or not. This way we can evaluate if

our system suggests items accurately.

Figure 1.1. Share of time spent per day with major media by US adults (in %). Source: [12]

Thirdly, we need to create a mobile quiz game which will serve as a base for testing both of the

systems to be implemented. It should be question-answer based game which will allow players to

answer movie based question generated by the system, create their own set of question to challenge

friends and to receive quizzes from other players. Additionally, gamification elements should be

implemented to encourage users to play more, as the more user plays the more info we can gather.

Finally, we have to evaluate the performance of our systems and provide the end results at the end

of our project no matter if our system was successful or not.

Chapter 2. Literature Review

In this chapter, the existing techniques and methods that have been considered and used in user

profiling, recommender systems and gamification are presented. Additionally, existing mobile

games are reviewed in order to estimate how suitable they are for the purposes of the project.

Section 2.1 discusses user profiling, its contents and techniques and current trends. Section 2.2

describes recommender systems, existing algorithms and techniques. We briefly discuss

gamification methods in Section 2.3. In Section 2.4 we review mobile games that can be found in

Play Market today, their advantages and disadvantages and how design decisions which were used

in these apps can be applicable in our context.

2.1. User profiling

This section presents user profiling, its contents, various methods and techniques that are used in

user profiling. Current trends in user profiling are also described, as well as the perspectives and

benefits of using user profiling.

2.1.1. User profile contents

In general, user profile is a representation of valuable information about a user that is important

for the domain we are currently considering. The contents of user profile can be different, such as

interests, knowledge, skills, goals, behaviour, preferences, etc. Let’s discuss the contents of user

profile thoroughly:

Interests. User interests are typically one the most important parts of the user profile in

information retrieval systems. Interests can vary from news topics or document topics till

hobbies-related topics and search queries. It could be classified as a short-term interest if

user reads topic only during Olympic Games or long-term ones, such as exchange rates.

For example, the system introduced in [5] learns about user’s interests by considering

his/her daily news topics and classifies recent events as short-term interests and general

preference for news as long-term ones. The most common representation of interests is

keyword based models where interests are shown as weighted vectors of keywords. It is

widely known in the area of Information Filtering or Information Retrieval. As an example,

let’s consider the system described in [6]. The system determines user’s interests by

identifying what kind of Web-pages user generally visits to build a hierarchy of tags and

categories. Each page is described as a vector of tags with corresponding weights

depending on the importance of the tag. The system then builds a hierarchy of categories

which looks like a tree, where root represents a general category of interest and leaves

represent internal nodes (specific interests) with corresponding weight extracted from web-

pages.

Knowledge and Skills. The background knowledge and skills that user has about the

application domain are very important. For example, in tutoring systems user’s previous

knowledge regarding the subject is essential to provide help to the user. This knowledge

can be described differently, however, the most common way to do it is to model the

knowledge using various values. It could be just binary (knows or does not know),

qualitative (good – average – bad) or quantitative, where probability is used to represent a

value. In [7] researchers consider user’s previous knowledge regarding Unix in order to

create a model which will adapt to the user. They classified users with respect to their

knowledge namely novices, beginners, intermediates and experts. Based on the knowledge

level their system presented user appropriate information, stretching and adding more

details for the beginners and shortening some things for experts who do not require some

basic information.

Plans. Plans show what purpose user wants to achieve while using the application. If user

is working with calendar app his/her goal is to manage or schedule events, if he/she is

browsing the web, then user’s aim is to gather particular information. Commonly, plan

recognition technique is used whose aim is to analyse user’s input tasks and try to find all

possible plans how this task can be explained. Armentano and Amandi [8] used plan

recognition technique in interface agents to analyse user’s plans. They took user’s input

and tried to analyse it using their technique. The system tries to find some evidences and

causes in the explanation that user gave. For example, they took an explanation: “Jack went

to liquor store. He pointed a gun at the owner” as an example. While analysing the first

sentence system will conclude that Jack’s goal is to shop at liquor store based on an

evidence that he went to liquor store. However, when it starts parsing the second one, the

system will change its mind and decide that Jack’s goal is to make a robbery. Based on

these decision system builds a profile where it knows what user wants to achieve.

Behaviour. Behaviour is another important part of the user profile. If it is repetitive, then

this pattern can be used by any adaptive system, for example, to adapt web page

accordingly. Behaviour is widely used in e-commerce to analyse customer’s actions [9].

For example, “when buying oats, Harry usually purchases milk” and “On weekends, Harry

spends more than 50 pounds”. After analysing this behaviour, the system can create a user

profile based on this behaviour rules. Now, as pointed in [9], we know that user likes milk

when buying oats, and likes to spend more than 50 pounds on weekends.

Individual Characteristics. Sometimes, personal information is needed to create user

profile. Mainly, individual information consists of gender, age, address, marital status and

other personal data. Commonly, this type of data is collected explicitly upon filling in some

forms or upon registration. As an example, this information can be used by travelling

agency to suggest you the best place to travel according to your marital status, age and

number of children.

Other information. User profile can also have data regarding interaction preferences which

describes the pattern of usage of an application. Additionally, contextual information is

collected to create user profile, which shows data that surrounds the user like temperature,

humidity, noise, etc. It is possible to collect information to create not individual, but group

profiles to deliver suggestions to a group of users [1].

2.1.2. Obtaining user profiles

After discussing the contents of user profiles, we can move on to creation of user profiles. This

section describes how user profiles can be build and how the data can be gathered from the user.

Basically, according to [1] and [10], obtaining user profile can be categorised as explicit, implicit

or hybrid user profiling. In this section we describe all of them.

Explicit user profiling. This is the simplest way of gaining required data from the user.

Usually, this type of data is gathered by filling in the online forms or surveys. Most of the

time, the information collected in this way is user’s age, gender, birthday, job title, hobbies,

etc. However, users often are not interested in revealing their private information or it could

happen that form filling is tedious so he/she tries to avoid it. This means that accuracy of

such type of profiling degrades over time [10]. For example, in [11] reader can explicitly

indicate which sections of newspaper he likes to read, choose topics of his/her interest,

such as basketball, he can also rate topics as interesting or not during his/her reading

process.

Implicit user profiling. As users are not willing to provide any private data the most

obvious approach to gather information is to observe actions undertaken by them, record

and log these actions, and identify patterns. According to [1], in order to create user profile

from actions, there are conditions that have to be met. The behaviour of the user must be

repetitive, if there is no repetition, behaviour pattern cannot be identified. As an example,

Godoy et al. [11] observes and estimates the amount of scrolling, reading duration and

whether the web page was added to bookmarks or not in order to generate user’s browsing

behaviour and to create his/her personal profile. Hybrid user profiling. This approach into user profiling combines both implicit and explicit

user profiling techniques [10]. It considers both the static data gathered about the user

(explicit user profiling) as well as retrieves user’s behavioural information (implicit user

profiling). As it takes into account the advantages of both methods described above, it

makes the user profiling more efficient by maintaining the accuracy of temporal data (as

data is updated temporally).

2.1.3. User profiling techniques

Obtaining a user profile is not a simple task, so there are a lots of ways how to do it properly based

on a domain. In this section, we describe current popular techniques to generate user profiles.

Generally speaking, these techniques are coming from the areas of Data Mining, Machine

Learning or Information Retrieval. We briefly discuss widely used user profiling techniques and

present some examples of their usage.

Association rules. This is a Data Mining technique used to identify patterns from data.

These rules were used to create user profiles in e-commerce area [9], however, they were

applied in other areas as well. An association rule is a rule which shows a connection

between a set of objects in a particular domain, such as one object implies another one or

multiple objects occur together. As an example, let’s consider research undertaken by [9].

Researchers were using e-commerce domain in order to generate association rules. They

created factual profile, which had user’s age, gender, name and his transactional profile,

which consists of customer’s purchases of particular product in a certain shop. Both of

these helped them to discover association rules for a given customer. Figure 2.1 shows

association rules discovered by Adomavicius et al. [9]. For instance, rule 1 shows that

customer named John Doe usually purchases product names as LemonJuice in a store

called RiteAid. The percentage shown in a row implies that in 95% of the cases when he

buys LemonJuice, he purchases it particularly at RiteAid. The second one, 2.4%, means

that among all John Doe’s transactions purchasing LemonJuice at RiteAid happened in

2.4% cases. Same logic applies to every other product found in Figure 2.1. The problem

which may arise is that sometimes there could be irrelevant rules, which could confuse the

system and may not represent user profile accurately. As pointed out in [9], one way of

dealing with such problem is to define some constraints on the types of rules of interest

before the rule discovery stage. This will reduce the number of irrelevant rules leading to

a better user profile.

Case Based Reasoning. It is a technique which solves issues by remembering previous

experiences. In this technique, every situation is represented as a case. If new situation

Figure 2.1. Association rules discovered which describes customer’s behaviour. Source: [9]

arises, it takes into account all suitable cases from previous issues that were already solved

by the system, which match current problem and adapts solutions thatå were used there to

solve the new problem. For example, in [11] this technique is widely used to generate user

profiles in a web-browsing domain. Each case records characteristics of a document read

by the user on the web, which gives opportunity to a reasoner to identify its topic based on

previous readings. These topics, generally, represent user’s interests. Using this approach,

we can categorize new topics using previous ones into specific categories, assuming that

similar docs can have similar topics. Although, this technique has its own advantages, it

has some limitations as well. The problem is that some documents in [11] are classified as

under the best matching topic, however, they can belong to one or more topics in some

cases. This may cause inappropriate creation of a user profile. Keyword tagging. We mark content using some kind of descriptive terms, which we call

tags or keywords. This is a common way of organizing content for filtering, search or

navigation. Organizing content in such way has earned a name “tagging” and is widely

used in different domains [13]. Elke Michlmayr et al. used keyword tagging technique in

their work [14], where they presented algorithm for profile construction using search

history and bookmarks of the user. In order to construct a user profile out of keywords that

user has used, they aggregated his/her interests according to their intensity. The most

obvious way to build it is to count the occurrence of keywords. The result of this approach

is a list of tags or keywords with corresponding number which resembles the weight of it,

which is shown in Table 2.1. The bigger the weight (the number), the more interested is

user in this tag. The main benefit of this technique is that it is easy to implement and fast.

However, is has some drawbacks, such as these tags sometimes are not fully specific (for

example the keyword web is quite general). Additionally, tags in a created profile are

sometimes unlinked ones and it can create misunderstandings.

Neural networks. Artificial neural network is a technique which is designed to simulate

biological neural system in a computer environment. As in a human nervous system, in

neural networks there are a lot of nodes that are connected with each other using weighted

connections [15]. The weights in these connections are initialised to small random values.

When some input and output values are given to the network, it “learns” by adjusting those

weights to represent the connection between input and output to the best possible values.

If enough input and output values are provided to the network, it can develop a model to

predict output values for a future inputs. This kind of technique is excellent for learning

sequences of operations and often used in stock market analysis or sales predictions. In

[15] researchers created a website logic which tracks the categories of items which user

visits in a sequence. They used collaborative filtering approach (which will be discussed

in a next section) to analyse previous users’ sequence of actions to predict what current

user might be looking for. Overall, their results were quite nice showing that in 49% of

their predictions consisted “very useful” links, while only in 7% of the time users felt links

were all useless. However, in case of this technique most of the time developers choose to

Table 2.1. List of tag ranked by their number of occurrence. Source: [14].

use general model to represent user. It means that some specific users won’t benefit from

the technique, however, it can be solved by creating individual profiles and collecting some

additional information about a user. Let us explain how it works in a more detailed way.

As shown in Figure 2.2, neural networks have highly connected structure of layers of

neurons that are called neural units. #1 through #5 shows the input data which is passed to

input layer. The data goes to hidden layer after that, which is connected with the input via

weighted connection. The weighted sum in each node of hidden layer is passed through

non-linearity function and then it sends the data forward to the output layer. For example,

if we want to predict the weather conditions, we will provide the day of the week, the

season and temperature as an input, while the predicted output would be whether it is going

to be raining or not.

2.2. Recommender systems

Recommender system will evaluate how good our user profiling algorithm is and, it will represent

how good our application is to the users by providing them correct suggestions. This section will

describe what recommender system is and what is it used for, what kind of approaches there are

for recommender systems and possible applications of recommender systems.

2.2.1. Overview

Recommender System (RS) is the tools and techniques which are used to provide appropriate

suggestion of items to the user [16]. Provided suggestions are related to some kind of decision

making process, such as what movies to watch, what music to listen to, or what book to read. In

this context, movies, music and books are called “items” by the recommender system. “Item” is

the general term which is used to mark the things that are recommended to users. Normally, the

system is focused on a specific type of item (e.g., movies or books) and all of its design and

implementation are customized in a way to provide best possible suggestions.

Basically, personalised suggestions are offered to the user as a list of items with a ranks. During

the ranking, system tries to predict what items are the most appropriate and accurate, based on a

Figure 2.2. Structure of an Artificial Neural Network. Source: [15].

user profile created before. So in order to perform this task we have to create user profile as we’ve

discussed previously.

The development of RS started from a very simple observation: persons usually try to rely on

recommendations given by others in doing daily and routine decisions, such as suggestion on

which book to read, recommendation letters when recruiting staff or movie review of a critic on a

website [16]. In order to simulate that kind of behaviour, the initial recommendation systems used

certain algorithms to evaluate previous recommendations made by a community to provide

suggestions to an active user. They were based upon items which were liked by other users with a

similar taste.

However, as time went by, techniques started to gain their shape and each of them started being

used for a certain purpose according to their functionality. The most popular implementations of

recommendation systems are provided below.

2.2.2. Recommendation System Implementation Techniques

In order for RS to predict which item is worth recommending, the system has to analyse and deduct

the utility of the items or, at least, compare items and then predict that item is indeed worth

suggesting to the user based upon the comparison. In order to demonstrate this prediction

procedure of RS, let us consider, as an example, non-personalised algorithm that suggests just the

most popular books. The purpose of using such an approach is that in absence of appropriate data

about user’s interests, it assumes that if the book is liked by many readers it could possibly be

preferred more than any other random book by an active user. This means that system predicts that

popular book’s utility for an active reader will be reasonably high.

Recommender systems have evolved and now, are complex systems, which have different

approaches of implementation. This section present current techniques used in recommender

systems.

Content based. In this technique, the system recommends items which are quite similar to

the ones user preferred before. This similarity is calculated based upon attributes of the

items compared. For instance, if user has rated a horror book positively before, then RS

can learn to suggest other books of this genre for him/her later. Pazzani et al. [17] in their

work used content based recommender system based upon user’s profile and item’s

description. According to their research, content-based recommender systems are perfect

when items are well attributed and well distinguished between each other. However, in the

absence of such attributes, no content-based system can provide proper suggestions.

Additionally, other type of systems, such as collaborative filtering ones can benefit from

content-based system, by filtering their results. For example, content based system can

easily distinguish lawyer joke from a chicken joke based on content, however, it might be

difficult for content based system to distinguish funny lawyer joke from other lawyer jokes.

In this case, collaborative filtering approach should be used which will consider which

lawyer jokes were preferred by other users. Collaborative filtering. In this approach, system recommends those items to the active user

which were liked by the others with a quite similar taste [16]. Similarity in this technique

is calculated using the history rating of the active user and its similarity with other’s. It is

the main reason why this technique is sometimes called “people-to-people correlation”

[18]. Collaborative filtering is the most popular technique in RS. One of the main

drawbacks of the collaborative filtering approach is the cold start. It describes the situation

when system gets new user, however, there is no information available about him/her, so

the system will not be able to provide proper recommendation. In order to provide good

prediction system should know a lot about active user. Knowledge-based. In knowledge-based systems there is a similarity function which

estimates how much the user needs match the suggestions provided by the system. So,

basically, similarity score calculated by the function can be interpreted as the utility of

suggestion for the user [16]. The main advantage of such systems is that they do not require

large statistical data about particular items or users. The only limitation it has is that it

should have an access to a database of items in order to make its calculations [19]. Community-based. In this kind of technique, friends of the user play major role, as the

recommendations are based upon their preferences [16]. A proverb “Tell me he who your

friends are, and I will tell you who you are” can be used to describe community-based

systems. This type of systems is often used in the context of social networks as they rely

on what your friends like. All the recommendations are based upon ratings that were

provided by user’s friends. Some reports [16] say that community-based recommendations

are not better than collaborative filtering ones, except in special cases, such as cold-start of

the system when user did not provide enough valuable information to compute suggestion

based on collaborative filtering approach. Additionally, researchers in [16] said that the

best solution is to combine both community-based and collaborative filtering approaches

as they will benefit from each other.

2.3. Gamification

In this section, we discuss the term “gamification” and how important it is in our research as we

are to develop a game. The overview of the term “gamification” is provided as well as everything

behind it.

The recent increasing trend in games and game technologies, served a catalyst to the growth of

research in this field. The most recent phenomenon which has been addressed is “gamification”, a

term which is used to deliver video game elements to a non-game domain [20].

Mobile games or games in general can be very powerful experience improving both motivation

and engagement. However, in recent years, this trend towards gamification of applications reduces

the complexity of a game to the very simple components (i.e. leader-boards, points, levels, etc.)

[21]. This results in decreased engagement among players. It does not mean that gamification does

not work. It means that in order to be truly successful, application should not only implement game

components, but game designs as well. In reality, games are not just a replacement for thoughtful

interaction or experience, they serve as a great frame for these kind of processes [21].

Sebastian Deterding [21] developed an achievement system for students which encouraged and

rewarded them to enhance their feeling of competence and progress. He pointed out that student’s

retention is improved when he/she is connected with faculty staff. To improve such connections,

he created special kind of collectible cards which student can achieve in interaction with a system.

Additionally, he created collaborative achievements which are given to freshmen if 90% of them

pass programming class, as University in which he works had difficulties in getting more than

85% of them to pass.

Proper real life games use full power of game design, while applications that are gamified

implement elements of games [22]. However, sometimes the difference between proper game and

application with game elements can be blurry. As an example, let’s take Foursquare which is an

international city guide that helps you find places to eat, drink, shop or visit around the world. It

has social features where you can share your opinion on places, as well as an achievement system

based upon how many places you have visited. You earn points when you check in to some place,

and you can challenge your friends based on the number of places you have been to. Is it a proper

game or a “gamified” app? Do you “play” it or “use” it? It all depends on your goal, focus or

perception, which are all subjective (negotiable). This raises a question: which elements are really

“game elements”? According to Reeves et al. [23] who identified “Ten ingredients of Great

Games”, there are several features which belong to “game elements” namely: avatars, which

represent the user; 3D environment; feedback; teams; context; reputation; ranks; marketplaces;

time pressure; etc. Each of them can be found in a non-gaming applications.

2.4. Current market overview

In this section we are going to inspect current movie based quiz applications, how their

implementation is applicable for our research and which features of current implementations can

we benefit from.

Let’s take a look at “Quizclash” by FEO Media AB which has more than 100 000 downloads [24].

It is a quiz based game, however, it covers different topics, from sports and medicine to history

and science. Undoubtedly, it has authentication system which we will also use in our system in

order to gather data and target recommendations. Another interesting thing that “Quizclash” has

is ability to add friends, which is very useful in our case, as we are going to challenge friends with

quizzes. In “Quizclash” one can play with friends using their unique login or can challenge random

players. Statistics is also one of their unique features which shows what topics are you good at,

your rank and score and percentage of wins, however, it is only available in premium version of

the app. The UI (user interface) of the question page is pretty simple and user friendly, as it only

has a space for the question itself and four buttons which represent the possible answers. Figure

2.3 shows the question page user interface. One more interesting thing about “Quizclash” is that

it is available in wide variety of languages, so developers have different apps with the same

functionality for different languages (Russian, Spanish, etc). Probably, this is done because of the

database and information variations for diversity of languages.

Another perfect example is “Trivia Crack” developed by Etermax which has more that 100 million

downloads in the Google Play Market [25]. Regarding questions and the UI of the page which

shows them, “Trivia Crack” is almost identical to the game we’ve discussed above (shown in

Figure 2.3). However, “Trivia Crack” is more gamified as it has rich achievement system and

profile representation with your own rank and level. As every other game of this type it has vast

variety of topics which are randomly chosen with roulette inside the app. It has messaging option

where you can chat with other players.

“QuizUp” by Plain Vanilla Games Corp is another option which is quite popular as it has more

than 10 million downloads [26]. It is even more socialised and gamified, as it has something like

a news feed where players can post what they want. However, it is quite different from the apps

we’ve discussed already. Topics are not chosen randomly; you choose them yourself upon

registration or sometime later from the list of topics. Then you can simply click on a topic and it

starts the game with a questions based on this topic and, of course, you are challenging some other

players from the system. You increase your level by answering the questions on each of the topics.

You can even follow your friends to see how they perform or which achievements do they have.

The UI for the questions is quite similar to the above mentioned games and shown in Figure 2.3.

As we can clearly see, all three games which we have been discussed here share common features

and even UI elements. The UI of the page which show the question is pretty much the same in all

of them; they all have social features like finding a friend, chatting with him and challenging him;

they all share achievement system which encourages players to play even more. They all have

wide variety of topics which are available for the users to choose.

Figure 2.3. Graphical user interface of question page. From left to right: «Quizclash», Source: [24]; «Trivia Crack», Source: [25]; «QuizUp», Source: [26].

Chapter 3. Requirement and Problem Analysis

To develop software which meets user’s demands we have to determine the actual needs of our

users first. The in-depth understanding of software requirements is key factor for software success.

This means that no matter how good the code is or how beautifully designed the software is, if it

doesn’t satisfy users in their needs it will disappoint them and it will make the end product useless.

This is why the document which lists requirements of the project has to be very specific, accurate

and clear.

The aim of this project is to develop a mobile game which will be a basis for two systems that are

going to be the main components of our research. Users will be able to play by asking each other

questions on movie topics. Our first system - user profiling system - will use this information in

order to build user interest profile which will contain what kind of movies our user is interested

in, whereas our second system – recommender system – will use generated profiles in order to

recommend movies to the user which he/she might be interested in. Thus the main object of our

project is to properly build user profile, then recommend appropriate movies to the user using

mobile application which is going to be developed.

This chapter provides analysis of the requirements of the system under development. It provides

the list of the stakeholders, system requirements followed by database requirements.

3.1. Stakeholders The stakeholders are to be specified in requirements analysis. Stakeholders are the people who

have direct or indirect interest in the system [27]. Let’s try to partition our stakeholders and provide

some description:

Gamers (i.e. users) – the ones who are going to play our game and the ones whose user

profiles we are going to generate. Developers – software engineers who work in mobile application development or in

recommender systems domain might be interested in the game itself, either to use it as a

pattern for their own application or in the user profiling and recommender system.

Third-parties – third party companies or persons who might be interested in data we

gathered to use it, as it may contain useful information about users, for example, in

particular location. So they might use this information to better target their products or

services.

3.2. System requirements

In this section we are going to describe the system requirements. The system requirements can

be divided into functional and non-functional requirements. Functional requirements usually

define the capabilities of the system and what functions will it be able to perform. Others, non-

functional requirements, describe the qualities which are used to judge how well system

performs.

Let us present the format of the functional requirements that is chosen:

ID: The identification number of the requirement;

Description: The requirement itself (it’s description);

Explanation: An in-depth explanation of the requirement;

3.2.1. Functional Requirements

Functional requirement describes a piece of functionality of the system which is going to be

needed. Functional requirements are backed up by non-functional requirements, which introduces

some constraints on the implementation and design. The functional requirements of our system

are listed below:

3.2.2. Non-Functional Requirements

Non-functional requirements define how system is going to do all of its functionality, not what it

will do. As an example, such things as performance requirements, external interface

requirements, software quality attributes or design constraints can be considered as non-

functional requirements of the system. The most important general non-functional requirements

are presented below according to [28]:

Security: defines how well the system is protected against potential threats.

Table 3.1.Functional requirements of the system to be implemented

ID Description Explanation

FR01 Players must be able to sign up Player must be able to sign up using some kind of

form with login, email and password

FR02 Players must be able to log in Player must be able to log into the system using

login and password

FR03 Players must be able to log out of

the system at any time

Player must be able to log out of the system using

“Logout” button at any time

FR04 Players must be able to change

their password

Player must be able to change their providing their

old one as a proof of owning an account

FR05 The system must be able to

provide an error dialog in case of

error upon login or registration

The system must show an error dialog to the user

highlighting the description of an error if any input

data is wrong

FR06 Players must be able to create

their own questions

Player must be able to create his/her own set of

questions in order to challenge other players in

future

FR07 Players must be able to search for

another players

Player must be able to use search function to find

user using login

FR08 Players must be able to add

friends to their friends list

Player must be able to find and add another player

as his/her friend in order to challenge him/her

FR09 Players must be able to play

standalone game with a system

Player must be able to start a game with a system

without challenging any other player

FR10 System must be able to connect to

remote movie database and show

data

System must be able to remotely connect to a

database of movies in order for players

FR11 System must be able to gather

needed data from users to create

user profiles

System must be able to properly collect necessary

data from users and generate users interest profiles

out of it

FR12 System must be able to

recommend user a movie based

on his/her user profile

System must be able to generate a proper

recommendation based upon generated user

interest profile

FR13 Players must be able to challenge

other players

Player must be able to challenge other player using

the questions they created

Performance: describes the execution speed of the system.

Extensibility: describes the ability of the system to be extended or modified to meet

possible new requirements.

Usability: it describes how easy user can learn to operate the software, prepare his/her

inputs for it and interpret the potential output of the system.

Cost: shows the total cost of the project in terms of time spent on developing and its

overall price.

Reliability: defines the ability of the system to execute required functionality for a

specific period of time under specified conditions.

Portability: shows how easy the system (or its components) can be transferred from

environment to another.

The list is ordered by the priorities that are going to drive our development process. Security is

the main concern as the project is dealing with the data gathered from users. Next priority is

performance, since recommendations should be delivered to users in a sensible time, without

having to wait more than they are expecting. Extensibility is what we hope to achieve next, as

we want to be able to add new features to our systems (user profiling or recommender) if it will

be required. As we are working on mobile application usability is our next concern, as users

nowadays require nicely designed and usable programs. Cost is next priority, since it is a

university project we do not want the price of the project to be higher than expected. Reliability

is the least of our concerns, as our main objective is to test our systems on a mobile platform,

namely Android OS, it is acceptable if our application crashes sometimes. Portability is in the

end of the list because we are concentrating on a one mobile operating system and not planning

to port it in terms of our project. However, as most of our code responsible for the systems is

located on the server it is possible to port our application to another platform without many

problems. Non-functional requirements of our system are listed below:

ID Description Explanation

NFR01 The graphical user interface

should be easy enough to

understand for regular users and

with a similar style

The buttons, menus, layout and dialogs should

have similar style across the system. As players

will execute specific functions they should see

familiar UI elements.

NFR02 The system should deliver clear

and detailed notifications to the

user with detailed messages

The system should deliver notifications to the

players with detailed feedback on operation

performed. If something requires player’s

attention an appropriate notification should be

displayer with the message

NFR03 The system should not have bugs

and error. If any error occurs, user

must be informed

The system should be tested and possible bugs

should be solved before release. If any error

occurs, a dialog with error message should appear

informing user of any wrong operation

NFR04 The system should be able to run

on all devices with Android OS

The system should be able to run on all Android

OS devices, such as smartphones, tablets which

use Android operating system

NFR05 The system should be able to

request login and password for

each user account

The system will not display any functionality

unless user logs into his/her account using unique

login and password

NFR06 The system should be able to

connect to remote server to get

necessary data

The connection between the system and remote

server should always be stable in order to get

necessary data

3.3. Database Requirements

This section describes the database requirements of our project. Since we are going to have two

types of databases, namely: movie database, which consists of movies and all the corresponding

information about them (title, release date, cast, etc.) and our own database which will store the

user data, questions and the user profiles which are going to be generated during the gameplay.

3.3.1. Movie Database Requirements

As the project is solely based on the movie data, we need quite detailed database which contains

as much information about the movie as possible. Not only it should be detailed and structured,

but it also must be easily accessible. List of the movie database requirements is shown below:

3.3.2. Local Database Requirements

The second type of database we are going to have is a database which stores user data, list of

questions and most importantly, the user profiles and preferences. List of our requirements for the

local database is provided below:

NFR07 The system should be able to have

fast response times

The should be able to execute all of its operations

as fast as possible. The player shouldn’t wait for

any operation take place longer than expected. In

case something needs time to be executed loading

dialog should be displayer to inform user of

execution

NFR08 The should be able to accept new

functionality and features

The system should designed in a way that any

necessary features could be implemented in an

easy way and added to the source code

Table 3.2. Non-Functional Requirements of the system

ID Description Explanation

MDR01 The movie database should be

easy enough to access

All the required information should be accessible

without significant limitations or constraints.

MDR02 The movie database should be

detailed and contain significant

amount of information about any

particular movie

Each particular movie taken from the database

should contain enough data, such as unique

identifier, title and overview.

MDR03 The movie database should be

able to categorise movies by

genres

Each movie should have a genre attached to it and

movies should be able to be sorted by genres

MDR04 The movie database should be

able to have directors of the given

movie

Each movie should have list of directors

MDR05 The movie database should be

free to use

In terms of our project we are not looking for the

solutions which require payment or donation in

order to make them work

Table 3.3. Movie Database Requirements of our system

ID Description Explanation

LDR01 The local database should be able

to have enough storage to hold all

the required tables and files

The storage should be large enough for all the

questions, user profiles and user data to be stored

especially with a huge amount of users playing

the game

LDR02 The local database should be

managed by any functional

database management system

As we are going to do a lot of operations on our

database we need to have decent database

management system already in place

LDR03 The local database should be

secure and protected from

potential threats

Since we are working with user data it highly

important to provide high level of security in

order not to leak important data

Table 3.4. Local Database Requirements of our system

Chapter 4: Design and Implementation

This section of the project describes the design and implementation of our system and its

components such as mobile application, web-server and both user profiling and recommendation

systems according to the requirements defined in Chapter 3. The design part is treated as the

foundation of the project development as it marks the start of development process. The

implementation is considered to be an actual execution of the design part. In this section we will

discuss the architecture behind our system and mobile application, decide which user profiling

technique will be used by our system and what kind of recommendation system algorithm will be

implemented. The data structure of our system is mentioned, as well as the security issues. At the

end of each section we will briefly discuss the implementation of each of these elements.

The mobile game which is going to be developed will give users the ability to play with each other

by asking questions based on movies. Users will have an ability to start single player mode, in

which they are going to play against the system, challenge their friends, where system is going to

generate the same questions for you and your friend, or create their own custom set of questions

and send them to a friend. Custom quiz created by user could be sent to a friend as a challenge,

giving our users flexibility by not limiting them only with questions stored in our system.

Analysing questions which were answered by players is going to provide us with the data needed

to make assumptions on the interests of our users. Such information will be used to create user

profiles which can be further used by our recommender system to make proper suggestions.

4.1. System As it was stated in Chapter 2, mobile devices are the most popular content consumption

instruments, this is why our project is going to be based on a mobile device. There are two main

competitive platforms available, namely Android and iOS. For our project Android is going to be

used as a platform of choice due to several reasons:

Bigger market share. According to [29], Android OS dominates the market with more than

80% share in 2015 Q2. As it can be seen from Figure 4.1, the market share of Android

Figure 4.1. Worldwide Smartphone OS Market Share from 2012 Q2 to 2015 Q2. Source: [29].

operating system is growing gradually year by year. Figure 4.2 shows us the current and

predicted market share of different mobile operating systems from 2014 to 2020. We can

clearly notice that Android OS holds and according to prediction will hold 80% of

smartphone shipments around the world. Bigger market share means more potential users,

which means more data to be collected. It gives us an opportunity to test and tune our

system better. Java language. The programming language which is mainly used for development on

Android OS is Java. In case of iOS Object-C language is used (Swift in latest versions).

Being familiar with Java programming language and not familiar with Object-C or Swift

made us choose Android OS.

However, there are some limitations and problems which Android OS has compared to iOS. One

of the major problems is:

Fragmentation. iOS only has a handful number of devices, as Apple Inc. only releases one

device each year, whereas Android operating system is widely used across all

manufacturers and they are releasing multiple devices each year with different

configurations. This leads to a problem called “fragmentation”, when you have too many

different devices with various hardware options. During the development, it is essential to

take into account this fact, optimise the code and test it on as many devices as possible

before the actual release. Same problem applies to fragmentation among versions of

operating systems. iOS devices are updated as soon as new version of operating system is

available. However, it is not the case with Android, where each manufacturer is responsible

for updating their own device. Figure 4.3 shows us current situation in fragmentation for

iOS and Android. We can clearly see that iOS has less fragmentation which leads to an

easier development process and less problems with optimisation.

Our system will consist of several parts. The client, the movie database, our server and the API

(Application Programming Interface) which will be used in order to connect everything with the

client. The client in our case is the mobile application which we will develop that is going to run

on a mobile phone of the user. An API in the code which is going to help us to use the features of

a movie database and our own server. A movie database is a service which is going to be used as

Figure 4.2. Current and predicted market share of worldwide smartphone shipments by operating system from 2014 to 2020. Source: [30]

a provider of information regarding movies and TV shows for our project. Our server is the

component that we will use to store the data gathered from users and to perform necessary

operations on this data. At the same time, our server will handle the user profiling and

recommendations, the code which is going to be stored and executed on our server. The overall

look of our system is shown on Figure 4.4.

4.2. The Client

The client is a mobile application which is going to be developed during the project. The

development will take place on Android operating system due to several reasons stated above. We

need a client which is built using the main components of Android OS, in order to benefit from

using this particular OS as much as possible. As the requirements of the users for beautifully

designed applications have risen dramatically, we have to consider the user interface as one of the

main factors of the client. The Java language is going to be used on our client as a main language.

The mobile game will be called “CineClash”.

In order to implement our mobile application on Android OS, we are going to use Android Studio

IDE [33], which is provided by Google Inc. for developers. The main components used in our

project are Activity [34] and Fragment [35]. Activity is a class which takes care of creating a

windows for user to interact with and which handles the UI. It is one the main classes which

handles all the interactions between user and the system [34]. Everything user touches or clicks on

is processed by “Activity” class in order to interact with the operating system. Listing 4.1

Figure 4.3. Operating system versions running on devices as of August 2016. Left – Android, Source: [31]. Right – iOS, Source: [32].

Figure 4.4. General overview of the system. Data flow between client, server and movie database.

represents the general overview of the code which is responsible for creating Activity class. The

second component of Android that we are going to use is called “Fragment” which represents a

small portion of UI. Each “Fragment” is attached to and controlled by “Activity” class. “Fragment”

can be used to build complex UI elements such as multi-pane UI and can be reused in multiple

Activities [35]. Each window shown to the user is either instance of Activity class or Fragment

class depending on the complexity of the elements shown. Listing 4.2 shows us a snippet of an

instance of Fragment which is responsible for login process. Additionally, Figure 4.5 showcases

the examples of user interface created during the project development. Appendix C contains all

the screenshots of the final version of the mobile application created.

4.3. Movie Database Design As we are building movie oriented mobile application we need a highly detailed movie database

in order to have all the required information regarding any particular taken movie. Building our

own is not an option as it is very time-consuming task and there are services available which

provide already highly populated database. The primary information we need about a movie is the

title, release date, genre, poster, it’s director and actors starring in that movie. However, the more

we can find, the better. This information is essential for us to instantiate proper user profiling and

provide appropriate recommendations. Most of the services which provide such information have

an API in order to access the data they store, however, not all of them are public or free to use. We

reviewed several services which provide such functionality. First one we tried was the most

popular one called IMDb, however, we have not found any information regarding their public API,

so we ended up using different solutions. TMDb API which is located at www.themoviedb.org is

Listing 4.1. Example of Activity class. Source: [34].

Figure 4.5. Screenshots of mobile application created during the project development. From left to right: Main Screen; “Start Game” screen; Question screen

the service that is going to be used in terms of the project as it provides full documentation on the

API and has all the required information about any movie or TV series. We can easily query any

necessary information using HTTP protocol’s [36] GET command. The service will send back

JSON object with all the information needed. Alternatively, OMDb API (www.omdbapi.com)

could have been used, which is also quite detailed and functional, however, it has one major

disadvantage. It requires a donation in order to acquire posters of the movies. As posters are going

to help us in a graphical user interface and are among the list of primary information we require

we cannot use OMDb API. Both of services require registration in order to get a unique key to

have an access to the service. After acquiring the unique key, it has to be used while querying the

desired information. Figure 4.6 resembles the entity-relationship diagram of the movie database

that we are connecting to, namely, www.themoviedb.org.

We are considering “Movie” and “TV Show” table to query all desired information. Each of them

are connected to several tables to get additional data. Each movie and TV show has an array of

genre IDs which are connected to the “Genre” table that stores the name of the genre and its unique

ID. Furthermore, movie has the list of companies which are responsible for the production of the

movie that links to the “Company” table. TV show has the same connection with the “Network”

Listing 4.2. Extract of Fragment class which is responsible for login process

Figure 4.6. Entity-relationship diagram of the movie database based upon our knowledge about their database and API documentation

table, which has the name and unique ID of each network responsible for the show. Same

connection applies to the cast row in each of the “Movie” and “TV Show” tables. Cast row is

connected to the “Person” table that stores ID of the person, his/her name and the job. The movie

database has “Job” table which consists of ID of the job and the name of it that connects to the

“Person” table through corresponding row.

As it was stated above, we are going to use HTTP’s GET [36] protocol. Android operating system

has built-in class called HttpUrlConnection that supports it. According to the rules defined in

Android, connection should be executed in a separate thread in order to keep main thread free for

handling the UI for the user. This is why AsyncTask [37] class will be used, which is a class that

allows the execution of a background operations and then publication of the results to the UI

thread. Listing 4.3 shows us a part of the code which is responsible for the connection to the movie

database and data retrieval from it.

4.4. Web Server Not only we need to have an access to a movie database, but we also have to have our own server

which will execute all the operations as well as store the user data. Web server has to support PHP

language in order to execute operations on the data gathered and support MySQL to store and

query information from our database. PHP is a server-side language [38, 39], which means that

PHP code can only be executed on machines (servers) with a PHP processor module enabled.

MySQL is database management system [39] widely used and is responsible for controlling and

querying the data from the database. In terms of our project we are going to use facilities provided

by Hoster.kz, as we already had a server available which was purchased before. However, any

other provider can be used as long as it provides a dedicated server with PHP and MySQL support

enabled.

Listing 4.3. Extract from AsyncTask class which is responsible for connecting and retrieving data from the movie database.

According to the Hoster.kz, their servers are based on HP ProLiant SL230s Generation8. Each of

them contains two six-core Intel Xeon processor with 128GB of RAM. During the project

development we will be using VPS (Virtual Private Server) provided by Hoster.kz whose

configuration is as follows: one core of Intel Xeon E5-2630v2 processor clocked at 2.6GHz,

256MB of RAM and 5GB of storage memory.

4.5. Interaction Design

As we have our own hosting which is capable of storing our data and we found service which

provides all the necessary information about the movies, we have to create a connection between

our mobile application and these two components. In order to establish connection between mobile

application and a web server we are going to use a client-server architecture with HTTP protocols

[36], which are used widely by web browsers.

The client-server architecture is an architecture which is used by many remote processors (client)

to get access to the services provided by host computers (server) [40]. The services running on the

server use ports which can be described as application identifiers. The client needs to know the

address of the server machine and the port number to connect to it. In this architecture client has

to know the server’s address and port, whereas server does not need to know anything about the

client’s address at the time of connection. Server passively waits for the connection to be

established by an active client. In our case mobile application will serve as a client while both

dedicated server and movie database provider will work as hosts. Figure 4.4 showcases general

view of our system and the way it communicates with our web-server and with movie database

service. As it can be seen from the Figure 4.4, our system is based on client-server architecture.

Users will be using mobile application developed during the project and will communicate with

our server, which will store user data, user profile and other necessary data, such as question and

answers to them. The communication between the application and server is handled using an API

developed. Application sends a request to the server using HTTP’s POST [36] method and server

sends back response with the data required. Each request contains a header which shows what kind

of method is to be invoked and all supporting arguments required for this method to run. The

server’s response contains result of the method invocation such as “success” or “failure” with

corresponding data required for the mobile application to continue running properly. The server

has PHP files that contains necessary functions for the database processes, user profiling and

recommendation operations. For example, player sends his/her login and password to the server

with a login request, after which server analyses the data and sends back response, which would

be “success” in case if login and password are correct or “failure” otherwise.

Listing 4.4. Extract of PHP code located on the server which is responsible for the login process

The connection between mobile application and the server will be done using HTTP’s POST [36]

protocol as it was stated above. The code responsible for connecting to our server is the same as it

was shown on Listing 4.3. However, the difference between them is that we are going to provide

“POST” as a request method to the instance of HttpUrlConnection class. The request will connect

to the PHP file located on the server, where scripts will be executed according to the request.

Listing 4.4 shows the example of PHP code located on the server. Next, according to the request,

it will connect to the MySQL database in order to insert, update or delete data. Every operation

executed in the mobile application which has to be processed on the server will be performed as

described above.

4.6. User Profiling As the main purpose of our project is user profiling and recommendation systems which are going

to work side by side helping each other, we are going to create both systems. In terms of our project

we are going to use hybrid user profiling approach in order to obtain user profiles which is a

combination of implicit and explicit user profiling [10]. This approach better corresponds to our

project’s context and it will make profiles more precise. Throughout the whole user profiling

process, we are going to use an assumption model which is based on the scores from -1 to 1. Figure

4.7 represents our model.

Implicit part. Technique which we decided to use in an implicit part of our user profiling

approach is keyword tagging [13, 14]. As we are dealing with movies keywords are going

to help us analyse user’s interests and make corresponding assumptions. Each question is

going to have a keyword attached to it and this keyword is going to help us to create user’s

preferences. Table 4.1 represents the schema of our questions table with an example.

Basically, we are using the title of the movie as a keyword in order to obtain additional

information about the movie from our movie database upon user’s answer. As it was stated

in [14], keyword tagging has a drawback where some keywords might be misleading and

too general. In our case, movies have quite distinguishable titles, which will eliminate this

drawback. The information we will query is genres of the movie and its directors.

Question ID Question Keyword

1 When was “Star Wars” movie released? Star Wars

2 Who directed “The Lord of the Rings”? Lord of the Rings Table 4.1. Schema of questions table with keywords

As user gets his question from our database, we query everything we need from our movie

database (genres, directors or networks). After he/she answers the question we check if the

answer was right or not. If it wasn’t right, we try to consider the director of the movie. We

assume that if the director is from the list of his/her favourites, it is possible that user hasn’t

seen that movie yet, so the movie might be a good candidate for recommendation. If it is

not the case, we assume that there is no connection between the user and the movie.

However, if he is right, we make an assumption that he is interested in this movie and

therefore in its genres and directors. This is why we create a new entry in our user

preference table with a status of 0, meaning that this is still an assumption which is due to

Figure 4.7. Model of the assumption to be made in the user profile process

be checked. In our project, status 1 will mean that the assumption on preference was

correct, while status 2 will mean it was incorrect. Same algorithm is applied to the quiz

creation by user. When user want to create a quiz, he has to create a questions first. During

the creation process user needs to choose a movie to which question is related to. Whenever

he submits a question, we analyse that movie and assume that user might be interested in

that movie, genres and directors. Table 4.2 showcases how our preference table will look

like.

User ID Movie title Genres Directors Status

1 Star Wars Action;

Adventure;

Sci-Fi

George Lucas 0

2 Lord of the Rings Adventure;

Fantasy

Peter Jackson 0

Table 4.2. User preference table schema with an example

Along with the preference we will start creating user’s personal profile based on genres

and directors. We are going to represent our user profile as an array of coefficients

corresponding to attributes. As soon as we create an entry in user’s preference table, we

create a new row in our user profiles. One of which is going to be based on genres, while

the second one will be based on directors. Based on how much time user spent on the

answer we are going to increment each genre’s and director’s score by value from 0 to 1

according to Figure 4.7. This way soon enough several genres or directors in user’s profile

will dominate and this will give us a picture of user’s interest. Table 4.3 shows us how

user’s genre profile will look like, while Table 4.4 represents user’s director profile.

However, it is not the final representation of the user’s profile which is going to be utilised.

We are going to discuss how we will use the profiles later in this section.

Action Adventure Sci-Fi Fantasy

User 1 12.3 9.4 3.6 4.3

User 2 2.5 6.8 14.7 11.3

User 3 6.4 12.3 8.1 6.9

Table 4.3. User's genre profile with corresponding weights

George Lucas Peter Jackson Zack Snyder

User 1 12.3 5.6 3.1

User 2 6.7 14.6 2.1

User 3 8.1 7.0 10.2

Table 4.4. User's director profile with corresponding weights

Explicit part. Next comes explicit part of our user profiling algorithm, which will use user

feedback model [1]. At this stage, as we already populated our user preference table a little

bit, we are going to ask users explicitly if they like the particular movie from the table or

not. If they answer positively, the status of the preference will be changed to 1 and the

weight of corresponding genres and director will be incremented by 1 point. In case the

answer is negative corresponding genre and director weight will be decremented by 1 point

and the status of the preference will change to 2.

We will create several tables, which will store the users, questions, answers, preferences and

profiles. All of them will be stored on our server and managed by MySQL database management

system. Our main table will be “Users” which will store the information about our users, such as

their e-mails, logins and encrypted passwords. Almost all the tables are connected with this table

by the unique ID of the user. Figure 4.8 depicts entity-relationship diagram of our database stored

on our server.

As it was stated above, every time user makes an action related to the movie we analyse the genres

and directors of the movie. Next thing we do is populate our database with gathered data. PHP

code written specifically for this purposes is going to help us with this problem. Small portion of

database tables are shown on the Figure 4.9.

Figure 4.8. Entity-relationship diagram of the database stored on our server

Figure 4.9. Extract from the database from our server. Top – User’s preferences; Left – User’s director profile; Right – User’s genre profile

4.7. Recommender System

Prior to creation of the systems, we’ve investigated the key factors which drive people to watch

movies or TV shows. Several researches have done much work in that field by conducting both

quantitative and qualitative studies. Apart from psychological factors, such as mood, opinion or

attitudes researchers found out that there are other factors which have great impact on movie

attendance decision making. For example, in [41] researchers concluded that the main goal of

viewers is to get new experience that they have never experienced in everyday life before.

Respondents pointed out that there are two factors which affect their decision. The first one is the

story behind the movie, which can be explained as a plot of the movie, background and music that

develop certain atmosphere. The second factor is more trivial one, which is movie’s genre, director

and actors. All participants in the study [41] prefer or dislike certain types of genres, and they gave

great attention to director of the movie. The actors were the least of their concerns. From this

knowledge we can conclude, that genres and directors are the main factors in choosing movie to

watch. Furthermore, to strengthen that argument, we’ve reviewed the study undertaken by Desai

et al. [42]. They studied how genre familiarity works together with the popularity of actors and

critics’ reviews. They concluded that if genre is familiar to the end viewer, actors has no impact

on their decision. However, if the genre is not familiar to the viewer, actors are going to play

important role in a positive decision to watch the movie. Same rule applies to the critics’ reviews,

where review has no effect on the performance of more familiar movie genres. By summarising

both of studies, we came to a conclusion that genres and directors play very important role in

viewers’ decision making.

We will build recommendation system which will be using the profiles in order to recommend

movies or TV shows. Content based technique in conjunction with collaborative filtering approach

is going to be used [16, 17]. Content based approach is perfect in our case as items (movies and

TV shows) in a movie database are well attributed and detailed. We will be using genres and

directors as an attributes in order to compare items. Content based approach can easily identify

which item to recommend using the attributes, however, it lacks the ability to determine which

item is better [17]. Here comes the collaborative filtering approach to help content based technique.

In terms of our project, it will identify the items which were liked by other users with the same

taste. This is how we are going to get the list of most popular items (in our case movies and TV

shows) based on the given attribute (genre and director).

Firstly, we have to normalise our data in order for all of the variables to be in proportion with one

another [43]. As we can see from Table 4.3 and 4.4 some of the weights could be a lot bigger than

another, so we have to first make them approximately equivalent to each other. That’s what we

call normalisation or standardisation, where all variables reflect meaningful relativity between

each other. There are many different normalisation techniques in Data Mining which could be

applied, such as Min-Max normalisation (which uses minimum and maximum values), Z-Score

normalisation (which uses arithmetic mean and standard deviation of data), Decimal scaling

(which uses logarithmic scale) and others [44]. However, in terms of our project we are going to

use Min-Max normalisation technique [43, 44] as it is very simple and at the same time quite useful

in transforming data values into standardised form of [0, 1]. Decimal spacing technique will fail

in our case as it relies upon logarithmic scale of data, for example, when one matcher has a score

in the range of [0,1] and another in the range of [0, 1000]. Z-Score normalisation will also fail as

it does not guarantee a common numerical range for the normalised scores if the original data is

not in the same range (has different matchers). However, min-max normalisation can be applied

even if the original values are not in the same range. Equation 4.1 shows Min-Max normalisation

formula which will be applied to our data.

𝑥𝑖,0 𝑡𝑜 1 =𝑥𝑖 − 𝑥𝑚𝑖𝑛

𝑥𝑚𝑎𝑥 − 𝑥𝑚𝑖𝑛

Equation 4.1. Min-Max normalisation formula. Source: [43, 44].

Where: 𝑥𝑖 = each value point i; 𝑥𝑚𝑖𝑛 = the minimum across values; 𝑥𝑚𝑎𝑥 = the maximum across

values; 𝑥𝑖,0 𝑡𝑜 1 = the value point i normalised between 0 and 1. Since we do not want to have value

of zero at any case (it will lead to no items being recommended) and value of 1 at any case either

(it will lead to absolutely all items being recommended) it is possible to choose an arbitrary range

of 0.1 to 0.9 [43]. So now, as we agreed upon the range, we can apply Min-Max normalisation

formula to our dataset in order to normalise it. Table 4.5 represents user’s genre profile after

normalisation. Table 4.6 shows user’s director profile.

Action Adventure Sci-Fi Fantasy

User 1 0.9 0.5 0.1 0.1

User 2 0.1 0.1 0.9 0.9

User 3 0.4 0.9 0.4 0.3

Table 4.5. User’s genre profile (Normalised)

User’s director profile will be normalised in the same way. After we normalise our profiles, we

can actually begin recommending items. Our recommendation model will be querying the most

popular movies in each genre of user’s profile, then querying the movies by which each director

is known for from the movie database. After we have a list of movies, we will remove the

duplicates, as there could be some movies which are popular in a several genres. So now, we have

a list of movies which could possibly be recommended to our user. We have to calculate

probabilities of the recommendation by using the coefficients from our normalised user profiles.

Let us give an example of how these probabilities are calculated. We will define an equation which

is going to be used to calculate these probabilities. We will consider “features” of our profiles, in

our case it will be the set containing genres and directors. Additionally, we are considering a set

of “tags”, in our case it will be an items from the list of all available genres and director. There are

as many set of tags as there are features. Our equation will look as follows:

𝑃(𝑢, 𝑚) = ∑ 𝜔𝑓( ∑ 𝜔𝑡

𝑡∈𝑇𝑎𝑔𝑠𝑓,𝑚

)

𝑓∈𝐹𝑒𝑎𝑡𝑢𝑟𝑒𝑠

Equation 4.2. Probability of recommendation equation

Where: P (u, m) – probability of recommendation of particular movie to the particular user, 𝜔𝑓 –

weight of a feature for a particular user u (weight of genre or director), 𝜔𝑡,𝑚 – weight of a tag for

specific feature of specific movie m for a specific user u. Weight of a tag can be acquired from our

normalised user profile according to the feature (genre or director), for example, a tag of the feature

“genre” can be “Action”. Weigh of feature can be calculated as 𝜔𝑓 =1

𝑛, where n represents the

number of features we are considering at the same time. However, how the feature is important to

George Lucas Peter Jackson Zack Snyder

User 1 0.9 0.1 0.2

User 2 0.3 0.9 0.1

User 3 0.1 0.2 0.9 Table 4.6. User's director profile (Normalised)

the user and how the weight of it can be different might be an interesting topic to consider in a

future work. Let’s consider an example in order to explain:

Movie One: Genres = {Action, Adventure, Sci-Fi}, Directors = {George Lucas};

Movie Two: Genres = {Adventure}, Directors = {Peter Jackson};

Movie Three: Genres = {Action; Fantasy}, Directors = {Zack Snyder};

Let’s calculate the probability of recommending each of these three movies to each user by using

an equation defined in Equation 4.2. As we are dealing with two features, namely Genres and

Directors, so 𝜔𝑓 (weight of a feature) is equal to 1

2 , 𝜔𝑡 (weight of a tag) can be taken from our

user profiles depending on a feature.

Movie One:

User 1 = 1

2(0.9 + 0.5 + 0.1) +

1

2(0.9) = 2.4;

User 2 = 1

2(0.1 + 0.1 + 0.9) +

1

2(0.3) = 1.4;

User 3 = 1

2(0.4 + 0.9 + 0.4) +

1

2(0.1) = 1.6;

Movie Two:

User 1 = 1

2(0.5) +

1

2(0.1) = 0.6;

User 2 = 1

2(0.1) +

1

2(0.9) = 1;

User 3 = 1

2(0.9) +

1

2(0.2) = 1.1;

Movie Three:

User 1 = 1

2(0.9 + 0.1) +

1

2(0.2) = 1.2;

User 2 = 1

2(0.1 + 0.9) +

1

2(0.1) = 1.1;

User 3 = 1

2(0.4 + 0.3) +

1

2(0.9) = 1.6;

After performing necessary calculations, we can clearly see that out of these three movies Movie

One is the best recommendation for User 1. Movie Two is the best option for User 2, while Movie

Three is the best match for User 3. Now we just need to sort the list by probability in descending

order and send it to the particular user.

The implementation part looks as follows: each time user opens an app it sends a request to the

server asking for recommendations. Server normalises the user profiles each time user sends a

request using the code shown on Listing 4.5. As our tables are indexed by an appropriate columns

normalisation for each user takes less than a second. Additionally, as we are using VPS in terms

of our project it means that we have superior performance than any regular hosting, so huge

amount of requests will not create any problems or delays.

After we normalise our profiles we request the most popular movies by each genre from the movie

database. In order to do this, we will use the cURL [45] library, which is built-in to our server. We

tried to do all the requests sequentially, however, it took approximately 20 seconds to finish, which

is not what we are looking for, as user doesn’t want to wait that much. This is why we will use

“curl_multi” command which is responsible for handling parallel requests. By using parallel

requests, we achieved much greater performance, reducing the speed from 20 seconds up to 2

seconds. This made the response times a lot more appropriate for users, so now they are able to

receive recommendations in a sensible time.

4.8. Security As we are dealing with data gathered from users it is highly important to provide proper security

for our information. Not only our server should be secure, but we have to provide safeness of our

data inside the database. According to Hoster.kz, which is our server provider of choice, they

provide their own protection against DDoS attacks, as they have an agreement with DDoS-Guard

company (https://ddos-guard.net/ru) which highly reputable in Russian networking market. So our

server is protected against such kind of attacks.

Additionally, we are protecting our database information with PHP language’s built-in methods.

Mainly, we create a protection for user’s passwords. Whenever user signs up into our application,

he has to provide a login and password. This password is processed through PHP’s “sha1” method,

which generates a hash code for the password to encrypt it using Secure Hash Algorithm 1 [46].

So basically, if someone gets an access to our database, he won’t be able to get any passwords of

the users.

In general, all the main security issues are handled by our provider, as it guarantees the safety of

their servers.

Listing 4.5. Part of the PHP code responsible for normalisation.

Chapter 5. Evaluation and Testing This chapter presents the evaluation and testing of our project. The first two sections show the

testing and evaluation of the requirements (functional and non-functional) of the project. The third

section presents both quantitative and qualitative evaluation of our recommendation system. The

last section is a summary of testing results.

5.1. Functional Requirements In order to test if our mobile application meets the functional requirements defined in Chapter 3,

we used integrated into Android Studio IDE [33] testing tools to set up and run the tests. Therefore,

we tested all elements of the application during and after the implementation.

The table below shows the test cases undertaken to evaluate the correct functionality of the

application. Each test case has its own unique identifier, followed by the functional requirement it

tests.

Test Reference TS-01

Test Requirement FR01

Test content Checks whether application allows user to sign up

Input User fills out the sign up form, using the login, email and password

Pass criteria Application displays a progress dialog and then redirects user to the

main screen

Test Reference TS-06

Test Requirement FR06

Test content Checks whether user is able to create his/her own questions

Input User fills out the question form, choosing the movie, inputting the

question text and choices

Pass criteria Application displays a progress dialog and then saves the question to the

database

Test Reference TS-08

Test Requirement FR08

Test content Checks whether user is able to add other users to his/her friends list

Input User types in the user name he/she wants to find and then clicks on

corresponding button to add user to the friends list

Pass criteria Application displays progress dialog, checks whether user exists in our

system and sends friend request if user exists or error dialog otherwise

Test Reference TS-09

Test Requirement FR09

Test content Checks whether user is able answer questions in standalone game

Input User clicks on “Start Game” button and receives a question to play

Pass criteria Application displays progress dialog, queries a question from the server

and displays it to the user

Test Reference TS-12

Test Requirement FR12

Test content Checks whether application is able to recommend movies to the user

Input User starts an application, then application sends a request to get

recommendations

Pass criteria Recommendation system calculates probability of recommending a

movie to the user and sends a list of movies to the application to display

to the user

Table 5.1. Test cases of Functional Requirements

Table 5.1 represents the list of test cases which test our functional requirements. The full list of

tests is available in Appendix D. All the functional requirements were met during the

implementation, thus all the test cases passed successfully. Table 5.2 shows the results of

functional requirements tests.

5.2. Non-Functional Requirements In this section we check if non-functional requirements, which were defined in Chapter 3, are met

within the project. Here is the list of non-functional requirements that were met:

NFR01. The application offers easy and understandable user interface in all screens

throughout the application using the same style for all the menus, buttons and layouts.

NFR02. The application shows clear and detailed notifications to the user whenever he

makes an action. This was tested by navigating through all the screens and by testing all

the functions which were implemented.

NFR03. The system doesn’t have any bugs as it was tested during the beta-test of the

application.

NFR04. The application is able to run on all Android OS devices which run version 4.1 or

higher as it was built using Android Studio IDE [33] which is provided by Google Inc.

NFR05. The application requires the login and password from the user in order to use all

the functions. No one can use the application functions unless they provide correct login

and password.

NFR06. The application connects to the server using HTTP’s protocols as it was stated in

Chapter 4.

NFR07. The system has fast response times, the UI is fast to respond and recommendations

are provided in a sensible time as it was stated in Chapter 4.

NFR08. The mobile application and system are designed in a way that new features can

be added easily by creating new screens in application if it is required or by adding

necessary functionality to the user-profiling or recommendation systems on the server.

Test Reference Test Result Test Reference Test Result

TS-01 PASS TS-08 PASS

TS-02 PASS TS-09 PASS

TS-03 PASS TS-10 PASS

TS-04 PASS TS-11 PASS

TS-05 PASS TS-12 PASS

TS-06 PASS TS-13 PASS

TS-07 PASS

Table 5.2. Test case results of Functional Requirements

5.3. Expert Evaluation In the previous sections we discussed the evaluation of the application from the designer/developer

point of view. In this section we will provide application’s evaluation from the user’s point of

view.

Basically, there are two main expert-based evaluation techniques which can confirm if the

application meets the requirements of the user:

Heuristic evaluation – is a quick, cheap and easy method of evaluation of the user

interface design. The heuristic evaluation involves a small set of evaluators to examine the

user interface and judge its compliance with usability principles (the “heuristics”) [47].

Cognitive walkthrough evaluation – is a task-specific evaluation method, where it is

assumed that user prefers to learn how to use a system by accomplishing tasks, rather than

studying a manual. This method helps to identify usability issues at the design stage, before

coding phase has even begun [48].

As application development took its place during the summer period, it was not possible to do

heuristics evaluation with a teacher and students as evaluators. Thus, we decided to run cognitive

walkthrough evaluation instead, with an expert as an evaluator, since heuristics evaluation requires

from three to five evaluators.

5.3.1. Cognitive Walkthrough Evaluation A participant who does not have any knowledge in software development and design was chosen,

since he will be the most unbiased evaluator. He was asked to perform a cognitive walkthrough

evaluation of our application.

A detailed description of our mobile application was given to the evaluator along with a tasks to

accomplish. The user interface was shown to the evaluator to run the tasks. The results were

following:

Task Steps Followed

Sign up 1. Press on “Sign Up” button on the welcome screen

2. Fill out the form (Login, E-mail, Password)

3. Press on the “Register” button

Log In 1. Fill out the form (Login and Password)

2. Press on the “Login” button

Start Single Player 1. Press on the “Start Game” button

2. Press on the “Single Player” button

Add user to the friends list 1. Press on the “Friends” button

2. Type user’s login in the search field

3. Press “Add to Friends list” button

Create a question 1. Press on the “Start Game” button

2. Press on the “Custom Quiz” button

3. Press on the “Create Quiz” button

Get recommendation 1. Start the application

2. Wait for the recommendation list to load

3. Press on the recommended movie or TV show

Provide feedback on

recommendation

1. Start the application

2. Wait for the recommendation list to load

The expert evaluator didn’t find any bugs or errors in our mobile application as it was tested for

bugs in advance.

Evaluator’s general notes:

The evaluator noted that the application is nicely designed and is quite easy to use. However, he

stated that sometimes it gets boring, as there is no motivation to play further. So, we concluded

that some gamification elements are missing, such as levels, achievements or tournaments. He

stated that the questions that are already in the system are quite challenging, however, it would be

nice to add more questions based on popular and currently trending movies. Evaluator liked the

ability to play different game modes. Creating own quiz and ability to send it to another player is

a great idea as said by our evaluator.

Evaluator’s notes on application’s strengths:

Simple and easy to use UI;

Different game modes to explore;

Challenging questions;

Ability to play with friends;

Evaluator’s notes on application’s weaknesses:

Not enough gamification (levels, achievements, etc.);

Needs more questions on popular or trending movies;

5.4. Recommender System Evaluation

This section presents the evaluation of the recommender system which was implemented in

Chapter 4. In order to test our systems adequately and in unbiased way we submitted our

application into Google Play Market. Our evaluation period lasted for almost 4 weeks after the

submission. Both quantitative and qualitative evaluation of our systems and application will be

provided. Quantitative evaluation is based upon special metrics which are aimed to test how well

the recommender system works. Qualitative evaluation is done using the questionnaire, the link to

which is provided through our mobile application on a main screen.

5.4.1. Qualitative Evaluation

In order to do the qualitative evaluation, we created a questionnaire for our users. The link to the

questionnaire is provided from the mobile application in order for them to have easy access to it.

Qualitative evaluation is done in order to get some insight from the users, to get the answers on

questions like why and how. The questionnaire consisted of 9 questions. Some of them were asked

to understand how particular terms are interpreted by our users, others asked how important these

terms are to our users. The questionnaire which was created is provided below.

3. Press on the “Like” or “Dislike” button on the

recommendation

Send quiz to another

player

1. Press “Start Game” button

2. Press “Custom Quiz” button

3. Select quiz from the quiz list

4. Press “Send” button

5. Choose a friend to send to

Table 5.3. Steps followed during Cognitive Walkthrough Evaluation

Questionnaire

1. In general, when do you think a recommendation list is diverse?

When items in the list differ with respect to genres

When items in the list differ with respect to directors

When items in the list differ with respect to actors

Other

By asking this question, we want to understand, what features are more important to our users

(genres, directors or actors). Additionally, this question is aimed to understand what user

understands by “diverse” recommendation list. This is a multiple choice question.

2. In general, when do you think a recommendation is interesting?

When I have not seen most of recommended movies/TV shows

When I would have not discovered the recommended movie/TV show by myself

When the list fitted well to my expectations

When the list fitted poorly to my expectations

When the combination of movies/TV shows was nice

Other

In this question, we are trying to understand what is interesting recommendation from the user’s

point of view. It gives an ability to further inspect our application through following questions.

This is also a multiple choice question.

3. What would be a motivation for you to use recommender system which recommends

Movies or TV shows?

There are too many Movies or TV shows, I suffer from information overload

I don't want to spend time searching for something interesting to watch

I need a suggestion on what to watch, as I don't know myself

I would like to discover movies and TV shows I am unfamiliar with

Other

This question gives us an ability to understand the main reason why users would use a system

which can recommend movies or TV shows. So, basically, we can understand, what drives them

to play our game even more.

4. To what extend do you agree with the following: “CineClash” gave me diverse

recommendations?

1 2 3 4 5

Not diverse o o o o o Very diverse

This question gives us a knowledge of whether our mobile application gave diverse

recommendations to the user, after we understood what users considers as a diverse

recommendation in the question number 1. The user gives a rating on how diverse were

recommendations.

5. To what extend do you agree with the following: “CineClash” gave me interesting

recommendations

1 2 3 4 5

Not interesting o o o o o Very interesting

In this question we ask user to give a rating on how interesting were recommendation suggested

by our mobile application. As we understood what user considers as an interesting

recommendation in question number 2, this question can provide us an information about how our

recommender system performs in this context.

6. To what extend do you agree with the following: I would use online web service which

recommends Movies or TV shows

1 2 3 4 5

Definitely not o o o o o Definitely will

This question asks user to what extend he/she would use web service to get desired

recommendations. It gives an ability to understand would user use web service instead of anything

else to get recommendations on Movies or TV shows.

7. To what extend do you agree with the following: I would use mobile application which

recommends Movies or TV shows.

1 2 3 4 5

Definitely not o o o o o Definitely will

This question asks user to what extend he/she would use mobile application to get desired

recommendation. It gives an ability to understand would user use mobile application instead of

anything else to get recommendations on Movies or TV shows.

8. Did you like “CineClash” game?

o Yes

o No

In this question we simply ask user if he/she liked or disliked our mobile application. This will

give us a glimpse on how good our application is.

9. In either case, please explain why?

This is not an obligatory question, compared to all others. This question ask user to give his/her

feedback on our application, in case there is anything in particular to say.

5.4.2. Quantitative Evaluation During our evaluation process which lasted for almost 4 weeks, 60 unique users started actively

using our application. Due to the short period of the evaluation process, we couldn’t gather a

significant amount of users to test our application and systems. However, even the small amount

of users that we got provided enough data to make conclusions.

In recommender systems, for the end user it is important to receive an ordered list of

recommendations, from the best to the worst. Taking this fact into account, we can apply the most

classic information retrieval metrics to evaluate recommender engine: Precision, Recall and F-

Measure [49, 50]. However, in each particular case we can use different metrics of evaluation.

There are two main cases:

Recommending some good items. In this case we make an assumption that there is a large

number of items that could be recommended to the user, however, we select only the

limited number of them. The reason for that could be the lack of resources (time or money).

This means that some items will be missing from the end list. This is mostly the case in

systems which suggest media items to the user, such as movies, music or books [50].

Recommending all good items. A less popular case where we recommend all available

items to the user. This case is applicable to the systems where it is very important not to

overlook any available items, such as system which recommend scientific papers or legal

databases [50].

So, in our case, we will stick with “recommending some good items” case, since we are

recommending media items to the user. Typically, in this case, we are interested in the binary

rating, that is, either the item was selected or not. Let us present the equations which we are going

to use in order to evaluate our system:

𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =𝑝𝑟𝑒𝑓𝑒𝑟𝑟𝑒𝑑 𝑟𝑒𝑐𝑜𝑚𝑚𝑒𝑛𝑑𝑎𝑡𝑖𝑜𝑛𝑠

𝑎𝑙𝑙 𝑟𝑒𝑐𝑜𝑚𝑚𝑒𝑛𝑑𝑎𝑡𝑖𝑜𝑛𝑠

Equation 5.1. Equation calculating precision. Source: [50].

𝑅𝑒𝑐𝑎𝑙𝑙 =𝑝𝑟𝑒𝑓𝑒𝑟𝑟𝑒𝑑 𝑟𝑒𝑐𝑜𝑚𝑚𝑒𝑛𝑑𝑎𝑡𝑖𝑜𝑛𝑠

𝑡𝑜𝑝 𝑟𝑒𝑐𝑜𝑚𝑚𝑒𝑛𝑑𝑎𝑡𝑖𝑜𝑛𝑠

Equation 5.2. Equation calculation recall. Source: [50].

𝐹 = 2 ∗𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 ∗ 𝑅𝑒𝑐𝑎𝑙𝑙

𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑅𝑒𝑐𝑎𝑙𝑙

Equation 5.3. Equation calculating F-Measure. Source: [49].

Equation 5.1 and Equation 5.2 presents Precision and Recall equations respectively. The

“Precision” is a proportion of recommendations that are preferred recommendations, that is,

proportion of preferred recommendations to all recommendations [49, 50]. The “Recall” is a

proportion of preferred recommendations that appear in the top recommendations, that means, the

proportion of preferred items to all good items recommended [49, 50]. Equation 5.3 presents the

F-Measure, which combines both “Precision” and “Recall” measures and shows the overall utility

of the recommendation list [49]. F-Measure has a value ranging from 0 to 1, where 1 means the

best value and 0 is the worst value.

Based upon [51], Precision and Recall often conflict with each other, meaning that increasing the

recommendation list will result in better Recall, while Precision will decrease. In our case, we can

say that Precision plays bigger role, as we are showing fixed amount of recommendations to the

user according to “recommending some good items” approach, however, in evaluating

recommender system both metrics are crucial. This is why in F-Measure, both of these metrics get

equal weight [51], assuming that both of them are equally important to the evaluation.

According to [50], a certain approach should be used when multiple users are involved. Based on

this approach, we need to calculate the precision and recall at each recommendation list length N

for each user, and then calculate the average. This will give us the values of Precision and Recall

of our system, which then can be used in order to compute F-Measure. During the testing period

we made almost 1700 unique recommendations in total, showing 40 top recommendations at the

same time on the main screen according to “recommending some good items” approach. Now let’s

compute the Precision and Recall for the whole system in Equations 5.4 and 5.5 respectively. We

will define the number of users as N. Basically, it is going to be the mean of precisions and recalls

for each particular user.

𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛𝑎𝑣𝑔 =1

𝑁∑ 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛𝑛

𝑁

𝑛=1

= 0.69

Equation 5.4. Average Precision of recommendation system

𝑅𝑒𝑐𝑎𝑙𝑙𝑎𝑣𝑔 =1

𝑁∑ 𝑟𝑒𝑐𝑎𝑙𝑙𝑛 = 0.61

𝑁

𝑛=1

Equation 5.5. Average Recall of recommendation system

After we got the Precision and Recall for the whole system, we now can calculate F-Measure of

the system. It will be as follows:

𝐹 = 2 ∗0.69 ∗ 0.61

0.69 + 0.61= 0.65

Equation 5.6. F-Measure of recommendation system

According to [52], F-Measure is value which is used to compare recommender system between

each other, since it tries to combine both Precision and Recall values. So, basically, F-Measure

most of the time is used only in comparison to some other systems, or if we are trying to use

different algorithm for recommendations, the values of F-Measure could be used to compare those

algorithms between themselves.

5.5. Summary In this section we will discuss the results of our evaluation process of the recommendation system.

Both qualitative and quantitative approaches will be discussed.

5.5.1. Qualitative Evaluation Summary

Unfortunately, not all of our users decided to take the survey. We ended up having 20 users willing

to answer our questionnaire. The full results of our questionnaire are provided in Appendix B.

According to the results of our questionnaire, most of our users prefer their recommendation list

to be diverse in terms of genres and directors. This makes us believe, that our user profiling and

recommendation systems work correctly and according to the users’ preference. The results show

our users think that recommendation is interesting when they haven’t seen the movie or TV show

that was recommended to them or when they wouldn’t have discovered it by themselves.

Furthermore, the main motivation for them to use some kind of recommender system is that our

users don’t want to spend time searching for something to watch. The second main reason is their

unfamiliarity with a lot of movies and TV shows, so they want to find something new to watch.

Our users gave our mobile application rating 4.1 out of 5 in terms of diversity of recommendations

according to question number 4. We can clearly see the correlation between our recommendation

system’s diversity and user’s feedback. The result for how interesting were our recommendations

was 4.4 out of 5 according to question 5. This is a great mark for our recommendation system, as

it means that our users haven’t seen or been familiar with the movies or TV shows that were

suggested to them. In terms of what our users will use to get recommendations, the result for web-

service was 3.3 out of 5, while the result for mobile application was 4.3 out 5. It clearly means that

our decision to create mobile application in terms of our project was correct.

In terms of diversity of our recommendations, the standard deviation (SD) and variance (V) of our

data was equal to 0.97 and 0.95 respectively. This means that 15 out of all respondents were within

the standard deviation. When it comes to interesting recommendations, SD and V was 0.85 and

0.73 respectively, again, showing us that 17 of our participants are within SD.

Out of all respondents, 17 users liked our application, while 3 others didn’t. The main reason for

a dislike was the personal preference, meaning they did not like that kind of games. However, if

to correlate this data with the other answers we gathered, we can clearly see that even if they did

not like our app, they did like the recommendations that it provided to them, therefore, our

recommendation system as well.

5.5.2. Quantitative Evaluation Summary According to our quantitative evaluation process, our recommender system performed quite well.

We decided to use “recommending some good items” approach, based upon [50], as it the best

option when it comes to recommending media items. We defined three main information retrieval

metrics, namely, “Precision”, “Recall” and “F-Measure”. Each of them test recommender system

in its own way.

Our results showed us that our recommender system performed very good in different ways.

According to our tests, our recommender system got 0.69 in “Precision” score, which means that

69% of all recommendation were satisfactory. At the same time, the score for “Recall” was 0.61,

which means that 61% of all good (top) movies recommended to our user were preferred by

him/her. For example, researchers in [53] created a Facebook utility which can recommend

movies, and their result for Precision was 0.81, while Recall was 0.7, while having 973 users in

their system. In [54], researchers tried to improve the system introduced in [53] by using Hybrid

Recommender System, however, the Precision of their system was 0.75 while having the same

amount of users. Although the results of our system is worse, but the difference is in the timeframe

and the number of people involved in the research. Researchers in [53] and [54] had more time

and significantly more users. However, even in the short time we had, our system showed positive

results. It is the early stages of our system, so we can say that it is the problem of the cold start that

collaborative systems suffer from in the beginning. The systems will only improve over time.

After we calculated both of these values, we computed F-Measure of our recommender system,

which is equal to 0.65. This means that the utility of our recommender system is above average in

a positive way, as score which is above 0.5 means that recommender system acts successfully.

Chapter 6. Conclusion This chapter summarises the work done during the project, and the project’s overall contribution.

6.1. Project Contribution The contribution to the education and mobile developer’s community is addressed in this section.

A study and analysis of user profiling and its theory was made. The contents of user profile,

ways of obtaining them and techniques which are used to generate user profiles were

discussed and analysed.

An in-depth study of recommender system was provided. The theory behind the

recommender systems and the ways of implementing these systems with examples are also

discussed.

The overview of gamification theory is provided. Additionally, the main principles behind

gamification are analysed.

A study of the mobile operating systems was made, as well as, their comparison with the

advantages and disadvantages of both operating systems.

Tactics for solving common problems in Android application development were provided.

As a result, fully functional Android application was developed, which can be installed on

any device running Android version 4.1 and higher.

Proper evaluation was made, based upon functional, non-functional requirements and UI.

Additionally, the ways of evaluating the recommender system is provided, as well as,

evaluation of our own recommender system.

6.2. Work summary The main goal of this project was to study whether it is possible to use mobile games in order to

extract required information to utilise user profiling and recommender systems. In order to

accomplish this task, a mobile application was developed which was submitted to Google Play

Store for users to test. It gave our users ability to answer question based on movies and TV shows,

play against their friends and create their own set of questions to challenge other players. The user

profiling system was built, whose aim was to analyse every answered or created question and make

corresponding assumptions on user’s preference and create user profiles. Two different user

profiles were created for each user, that are, user’s genre profile and user’s director profile.

Furthermore, a recommender system was created, the goal of which was to use these profiles to

suggest proper movies and TV shows to the user. A proper evaluation and testing was done on our

application and systems in order to identify their performance and successfulness.

Our evaluation clearly shows us that our recommender system worked positively good. At

Precision rate of 69%, Recall rate of 61% and F-Measure of 0.65 we can conclude that our

recommender system performed at a decently high level. However, as we already know, good

recommendation system is based upon good user profiling system, which means that our user

profiling system did its job properly. Even though we didn’t have a lot of time to test our systems

thoroughly, our systems still showed positively good results. The system will perform even better

when we gather more user data. Thus, we believe, that the research done is quite useful for future

developers to explore, as it answers our main research question. Every evaluation made, every

result obtained leads us to the main conclusion, which is, mobile games can be and should be used

in user profiling and recommender systems.

Chapter 7. Future Work This section discusses the possible improvements that can be made in order to improve the

application and both user profiling and recommender systems.

Improvements in the mobile application Although our mobile application is quite nicely designed, there are some flaws in the contents of

the application, namely, gamification part. As it was pointed out by our expert in Chapter 5, the

game gets boring at some point, as it doesn’t give any motivation to play further. To improve this

part, future developers should consider adding more gamification elements to the game, such as

levels, achievements, rewards, etc. This will create a motivation for our users to play more, to

explore every possible ability to increase their level. Additionally, as it was said by our evaluator,

our application lacks questions based around popular and currently trending movies. Although it’s

not a big issue and just a matter of time which is required to populate the database of questions,

future developers might consider starting their database from these kind of questions first.

Improvements in the user profiling Our user profiling algorithm did its job as we expected, however, as we all know, nothing is

perfect. We did our user profiling based on genres and directors. As it was stated in Section 4.6,

not only these features are important for the end users. Future developers should also consider

adding actors, critics’ reviews or even analysing the general overview or plot of the movie in order

to create profiles. This will give developers even more detailed data regarding the user.

Improvements in the recommender system Even though our recommender system performed quite well, there is always room for

development. As recommender system works together with user profiling system, it depends upon

it. So, if future developers update our profiling algorithm, there is going to be more work for the

recommender system. However, the main interesting part of the system which might be addressed

by the future developers, is the weights, especially, the weight of the features (genres, directors,

actors, etc.) discussed in Section 4.6. Future developers should investigate how important each

feature is for the particular taken user and integrate it into our equation. This might give more

precise probabilities of recommending an item.

References

[1] Schiaffino, Silvia, and Analía Amandi. "Intelligent user profiling." Artificial Intelligence An

International Perspective. Springer Berlin Heidelberg, 2009. 193-216.

[2] Middleton, Stuart E., Nigel R. Shadbolt, and David C. De Roure. "Ontological user profiling

in recommender systems." ACM Transactions on Information Systems (TOIS) 22.1 (2004): 54-88.

[3] Baraglia, Ranieri, and Fabrizio Silvestri. "Dynamic personalization of web sites without user

intervention." Communications of the ACM 50.2 (2007): 63-67.

[4] Zhao, Z., Cheng, Z., Hong, L., & Chi, E. H. "Improving User Topic Interest Profiles by

Behavior Factorization." Proceedings of the 24th International Conference on World Wide Web.

International World Wide Web Conferences Steering Committee, 2015.

[5] Billsus, Daniel, and Michael J. Pazzani. "A personal news agent that talks, learns and

explains." Proceedings of the third annual conference on Autonomous Agents. ACM, 1999.

[6] Godoy, Daniela, and Analia Amandi. "User profiling for web page filtering." IEEE Internet

computing 9.4 (2005): 56-64.

[7] Boyle, Craig, and Antonio O. Encarnacion. "MetaDoc: an adaptive hypertext reading

system." Adaptive Hypertext and Hypermedia. Springer Netherlands, 1998. 71-89.

[8] Armentano, M., and A. Amandi. "A bayesian networks approach to plan recognition for

interface agents." Proc. Argentine Symposium on Artificial Intelligence. 2006.

[9] Adomavicius, Gediminas, and Alexander Tuzhilin. "Using data mining methods to build

customer profiles." Computer 2 (2001): 74-82.

[10] Kanoje, Sumitkumar, Sheetal Girase, and Debajyoti Mukhopadhyay. "User Profiling

Trends, Techniques and Applications." arXiv preprint arXiv:1503.07474(2015).

[11] Godoy, Daniela, Silvia Schiaffino, and AnalıA Amandi. "Interface agents personalizing

Web-based tasks." Cognitive Systems Research 5.3 (2004): 207-222.

[12] Statista. “Mobile is the New Media Star”, www.statista.com [Online]. Available:

https://www.statista.com/chart/2168/share-of-time-spent-per-day-with-major-media/ [Accessed:

May 18, 2016].

[13] Golder, Scott A., and Bernardo A. Huberman. "Usage patterns of collaborative tagging

systems." Journal of information science 32.2 (2006): 198-208.

[14] Michlmayr, Elke, and Steve Cayzer. "Learning user profiles from tagging data and

leveraging them for personal (ized) information access." (2007): 1-7.

[15] Silver, Daniel L. "Web-based User Profiling Using Artificial Neural Networks.", Acadia

University.

[16] Ricci, Francesco, Lior Rokach, and Bracha Shapira. Introduction to recommender systems

handbook. Springer US, 2011.

[17] Pazzani, Michael J., and Daniel Billsus. "Content-based recommendation systems." The

adaptive web. Springer Berlin Heidelberg, 2007. 325-341.

[18] Schafer, J. Ben, et al. "Collaborative filtering recommender systems." The adaptive web.

Springer Berlin Heidelberg, 2007. 291-324.

[19] Burke, Robin. "Integrating knowledge-based and collaborative-filtering recommender

systems." Proceedings of the Workshop on AI and Electronic Commerce. 1999.

[20] Deterding, Sebastian, et al. "Gamification. using game-design elements in non-gaming

contexts." CHI'11 Extended Abstracts on Human Factors in Computing Systems. ACM, 2011.

[21] Deterding, Sebastian. "Gamification: designing for motivation." interactions 19.4 (2012):

14-17.

[22] Deterding, Sebastian, et al. "From game design elements to gamefulness: defining

gamification." Proceedings of the 15th international academic MindTrek conference:

Envisioning future media environments. ACM, 2011.

[23] Reeves, Byron, and J. Leighton Read. Total engagement: How games and virtual worlds

are changing the way people work and businesses compete. Harvard Business Press, 2013.

[24] Google Play Market. “Quizclash”. www.play.google.com [online]. Available:

https://play.google.com/store/apps/details?id=se.feomedia.quizkampen.uk.lite [Accessed: June

19, 2016].

[25] Google Play Market. “Trivia Crack”. www.play.google.com [online]. Available:

https://play.google.com/store/apps/details?id=com.etermax.preguntados.lite [Accessed: June 19,

2016].

[26] Google Play Market. “QuizUp”. www.play.google.com [online]. Availabe:

https://play.google.com/store/apps/details?id=com.quizup.core [Accessed: June 19, 2016].

[27] Power, C. (2012). Lecture: “Lecture 1: Introduction to requirements engineering”,

Requirements Engineering module. University of York. 2012.

[28] Glinz, Martin. "On non-functional requirements." 15th IEEE International Requirements

Engineering Conference (RE 2007). IEEE, 2007.

[29] International Data Corporation (IDC), “Smartphone OS Market Share 2015Q2”,

www.idc.com [Online]. Available: http://www.idc.com/prodserv/smartphone-os-market-

share.jsp [Accessed: August 7, 2016].

[30] Statista. “Market share worldwide smartphone shipments by operating system from 2014 to

2020”, www.statista.com [Online]. Available: http://www.statista.com/statistics/272307/market-

share-forecast-for-smartphone-operating-systems/ [Accessed: August 7, 2016].

[31] Android Developers. “Dashboards”, www.developer.android.com [Online]. Available:

https://developer.android.com/about/dashboards/index.html [Accessed: August 7, 2016].

[32] Apple Developers. “App Store”, www.developer.apple.com [Online]. Available:

https://developer.apple.com/support/app-store/ [Accessed: August 7, 2016].

[33] Android Developers. “Android Studio”, https://developer.android.com [Online]. Available:

https://developer.android.com/studio/index.html [Accessed: August 7, 2016].

[34] Android Developers. “Activity”. https://developer.android.com [Online]. Available:

https://developer.android.com/reference/android/app/Activity.html [Accessed: August 7, 2016].

[35] Android Developers. “Fragment”. https://developer.android.com [Online]. Available:

https://developer.android.com/guide/components/fragments.html [Accessed: August 7, 2016].

[36] Hypertext Transfer Protocol, RFC 2616. [Online]. Available:

http://tools.ietf.org/html/rfc2616/ [Accessed: August 8, 2016].

[37] Android Developers. “AsyncTask”. https://developer.android.com [Online]. Available:

https://developer.android.com/reference/android/os/AsyncTask.html [Accessed: August 7,

2016].

[38] Bakken, Stig Saether, Zeev Suraski, and Egon Schmid. PHP Manual: Volume 2. iUniverse,

Incorporated, 2000.

[39] Welling, Luke, and Laura Thomson. PHP and MySQL Web development. Sams Publishing,

2003.

[40] Berson, Alex. Client/server architecture. McGraw-Hill, Inc., 1996.

[41] Herlina, Dyna. "Identifying Key Factors Affecting Consumer Decision Making Behavior in

Cinema Context: A Qualitative Approach." International Conference on Business, Economics,

Management and Behavioral Sciences. Held on: Jan. 2012.

[42] Desai, Kalpesh Kaushik, and Suman Basuroy. "Interactive influence of genre familiarity,

star power, and critics' reviews in the cultural goods industry: The case of motion

pictures." Psychology & Marketing 22.3 (2005): 203-223.

[43] B. Etzkon. “Data Normalization and Standartization”, November 6, 2011. [Blog Entry]. Ben

Etzkon’s Financial Resources Blog. Available: http://www.benetzkorn.com/2011/11/data-

normalization-and-standardization/ [Accessed: August 17, 2016].

[44] Jain, Anil, Karthik Nandakumar, and Arun Ross. "Score normalization in multimodal

biometric systems." Pattern recognition 38.12 (2005): 2270-2285.

[45] PHP Manual. “Documentation. cURL”. www.php.net [Online]. Available:

http://php.net/manual/en/book.curl.php [Accessed: August 20, 2016].

[46] RFC Base. “US Secure Hash Algorithm 1 (SHA-1)”. http://www.rfc-base.org/ [Online].

Available: http://www.rfc-base.org/txt/rfc-3174.txt [Accessed: August 21, 2016].

[47] NNG Group, “10 Usability Heuristics for User Interface Design”.

https://www.nngroup.com/ [Online]. Available: https://www.nngroup.com/articles/ten-usability-

heuristics/ [Accessed: August 23, 2016].

[48] Wharton, Cathleen, et al. "The cognitive walkthrough method: A practitioner's

guide." Usability inspection methods. John Wiley & Sons, Inc., 1994.

[49] Marcel Caraciolo, “Evaluating Recommender Systems - Explaining F-Score, Recall and

Precision using Real Data Set from Apontador”, May 4, 2011. [Blog Entry]. A Blog About

Scientific Python, Data, Machine Learning and Recommender Systems. Available:

http://aimotion.blogspot.co.uk/2011/05/evaluating-recommender-systems.html [Accessed:

August 24, 2016].

[50] Gunawardana, Asela, and Guy Shani. "A survey of accuracy evaluation metrics of

recommendation tasks." Journal of Machine Learning Research 10. Dec (2009): 2935-2962.

[51] Sarwar, Badrul, et al. "Analysis of recommendation algorithms for e-commerce."

Proceedings of the 2nd ACM conference on Electronic commerce. ACM, 2000.

[52] Powers, David MW. "What the F-‐measure doesn’t measure…." (2014).

[53] Mirizzi, Roberto, et al. "Movie Recommendation with DBpedia." IIR. 2012.

[54] Lekakos, George, and Petros Caravelas. "A hybrid approach for movie recommendation."

Multimedia tools and applications 36.1-2 (2008): 55-70.

Appendix A Questionnaire used in qualitative evaluation:

1. In general, when do you think a recommendation list is diverse?

When items in the list differ with respect to genres

When items in the list differ with respect to directors

When items in the list differ with respect to actors

Other

2. In general, when do you think a recommendation is interesting?

When I have not seen most of recommended movies/TV shows

When I would have not discovered the recommended movie/TV show by myself

When the list fitted well to my expectations

When the list fitted poorly to my expectations

When the combination of movies/TV shows was nice

Other

3. What would be a motivation for you to use recommender system which recommends

Movies or TV shows?

There are too many Movies or TV shows, I suffer from information overload

I don't want to spend time searching for something interesting to watch

I need a suggestion on what to watch, as I don't know myself

I would like to discover movies and TV shows I am unfamiliar with

Other

4. To what extend do you agree with the following: “CineClash” gave me diverse

recommendations

1 2 3 4 5

Not diverse o o o o o Very diverse

5. To what extend do you agree with the following: “CineClash” gave me interesting

recommendations

1 2 3 4 5

Not interesting o o o o o Very interesting

6. To what extend do you agree with the following: I would use online web service which

recommends Movies or TV shows

1 2 3 4 5

Definitely not o o o o o Definitely will

7. To what extend do you agree with the following: I would use mobile application which

recommends Movies or TV shows.

1 2 3 4 5

Definitely not o o o o o Definitely will

8. Did you like “CineClash” game?

o Yes

o No

9. In either case, please explain why?

___________________________

Appendix B

Questionnaire results taken during the evaluation phase.

1. In general, when do you think a recommendation list is diverse?

Answer #

When items in the list differ in terms of genres 13

When items in the list differ in terms of directors 15

When items in the list differ in terms of actors 8

Other 0

2. In general, when do you think a recommendation is interesting?

Answer #

When I have not seen most of recommended movies/TV shows 15

When I would have not discovered the recommended movie/TV show by myself 13

When the list fitted well to my expectations 7

When the list fitted poorly to my expectations 2

When the combination of movies/TV shows was nice 3

Other 0

3. What would be a motivation for you to use recommender system which recommends

Movies or TV shows?

Answer #

There are too many Movies or TV shows, I suffer from information overload 6

I don't want to spend time searching for something interesting to watch 14

I need a suggestion on what to watch, as I don't know myself 5

I would like to discover movies and TV shows I am unfamiliar with 13

Other 0

4. To what extend do you agree with the following: “CineClash” gave me diverse

recommendations

1 2 3 4 5 Avg. Standard

Deviation Variance

0 2 3 7 8 4.1 0.97 0.95

5. To what extend do you agree with the following: “CineClash” gave me interesting

recommendations

1 2 3 4 5 Avg. Standard

Deviation Variance

0 1 2 6 11 4.4 0.85 0.73

6. To what extend do you agree with the following: I would use online web service which

recommends Movies or TV shows

1 2 3 4 5 Avg. Standard

Deviation Variance

1 4 7 5 3 3.3 1.09 1.19

7. To what extend do you agree with the following: I would use mobile application which

recommends Movies or TV shows.

1 2 3 4 5 Avg. Standard

Deviation Variance

1 0 2 7 10 4.3 0.99 0.99

8. Did you like “CineClash” game?

Answer #

Yes 17

No 3

9. In either case, please explain why?

Reasons mentioned by respondents:

I do not like this kind of applications

Appendix C

The full list of screenshots of the mobile application developed in terms of the project called

“CineClash”:

Login Screen Registration Screen Main Screen

“Start Game” Screen Question Screen Game Results Screen

Create Question Screen Recommendations Screen Movie Information Screen

Profile Screen Rating Screen

Appendix D Functional Requirements testing described in Chapter 5.

Test Reference TS-01

Test Requirement FR01

Test content Checks whether application allows user to sign up

Input User fills out the sign up form, using the login, email and password

Pass criteria Application displays a progress dialog and then redirects user to the

main screen

Test Reference TS-02

Test Requirement FR02

Test content Checks whether application allows user to log in

Input User fills out the login form, using his/her login and password

Pass criteria Application displays a progress dialog and then redirects user to the

main screen

Test Reference TS-03

Test Requirement FR03

Test content Checks whether application allows user to log out

Input User click on the logout button

Pass criteria Application displays a progress dialog and then redirects user to the

login page

Test Reference TS-04

Test Requirement FR04

Test content Checks whether application allows user to change their password

Input User click on the “Change Password” button and types in old and new

passwords

Pass criteria Application shows progress dialog and then show success dialog if old

and new passwords are not equal

Test Reference TS-05

Test Requirement FR05

Test content Checks whether application is able to show error dialog upon login or

registration

Input User fills out the login or registration form

Pass criteria Application displays an error dialog in case the login or password was

incorrect or in case of registration if login was already taken

Test Reference TS-06

Test Requirement FR06

Test content Checks whether user is able to create his/her own questions

Input User fills out the question form, choosing the movie, inputting the

question text and choices

Pass criteria Application displays a progress dialog, adds question to the database

and accepts the question

Test Reference TS-07

Test Requirement FR07

Test content Checks whether user is able to search for another players

Input User types in the user name of another player

Pass criteria Application displays progress dialog, checks if user exists. Show

another player to the user it exists, error dialog otherwise

Test Reference TS-08

Test Requirement FR08

Test content Checks whether user is able to add other users to his/her friends list

Input User types in the user name he/she wants to find and then clicks on

corresponding button to add user to the friends list

Pass criteria Application displays progress dialog, checks whether user exists in our

system and sends friend request if user exists or error dialog otherwise

Test Reference TS-09

Test Requirement FR09

Test content Checks whether user is able to play standalone game

Input User click on “Start Game” button and starts to play

Pass criteria Application displays a progress dialog and starts to query a question

from the system

Test Reference TS-10

Test Requirement FR10

Test content Checks whether application is able to connect to the movie database

Input User click on recommender movie and opens information about the

movie

Pass criteria Application displays recommended movies to the user, he clicks on the

movie and application queries movie related data from movie database

and displays it to the user

Test Reference TS-11

Test Requirement FR11

Test content Checks whether application is able gather data from users

Input User plays the game, answers the questions or creates questions

Pass criteria Application detects user’s answer or created question, queries

appropriate data and sends it to the server creating user profile

Test Reference TS-12

Test Requirement FR12

Test content Checks whether application is able to recommend movies to the user

Input User starts an application, then application sends a request to get

recommendations

Pass criteria Recommendation system calculates probability of recommending a

movie to the user and sends a list of movies to the application to display

to the user

Test Reference TS-13

Test Requirement FR13

Test content Checks whether user is able to challenge other players

Input User opens a friends list, click on a user and hits “Challenge”

Pass criteria Application displays a progress dialog, sends appropriate data to the

server, if everything is okay shows success dialog, error dialog

otherwise