the effect of assessment on motivation in a virtual ...1471397/fulltext01.pdf ·...

Department of Science and Technology Institutionen för teknik och naturvetenskap Linköping University Linköpings universitet

gnipökrroN 47 106 nedewS ,gnipökrroN 47 106-ES

LiU-ITN-TEK-A--20/010-SE

The Effect of Assessment OnMotivation in a Virtual Reality

Based Serious GameDavid Lindh

2020-05-29

LiU-ITN-TEK-A--20/010-SE

The Effect of Assessment OnMotivation in a Virtual Reality

Based Serious GameExamensarbete utfört i Medieteknik

vid Tekniska högskolan vidLinköpings universitet

David Lindh

Handledare Karljohan Lundin PalmeriusExaminator Daniel Jönsson

Norrköping 2020-05-29

Upphovsrätt

Detta dokument hålls tillgängligt på Internet – eller dess framtida ersättare –under en längre tid från publiceringsdatum under förutsättning att inga extra-ordinära omständigheter uppstår.

Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner,skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat förickekommersiell forskning och för undervisning. Överföring av upphovsrättenvid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning avdokumentet kräver upphovsmannens medgivande. För att garantera äktheten,säkerheten och tillgängligheten finns det lösningar av teknisk och administrativart.

Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman iden omfattning som god sed kräver vid användning av dokumentet på ovanbeskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådanform eller i sådant sammanhang som är kränkande för upphovsmannens litteräraeller konstnärliga anseende eller egenart.

För ytterligare information om Linköping University Electronic Press seförlagets hemsida http://www.ep.liu.se/

Copyright

The publishers will keep this document online on the Internet - or its possiblereplacement - for a considerable time from the date of publication barringexceptional circumstances.

The online availability of the document implies a permanent permission foranyone to read, to download, to print out single copies for your own use and touse it unchanged for any non-commercial research and educational purpose.Subsequent transfers of copyright cannot revoke this permission. All other usesof the document are conditional on the consent of the copyright owner. Thepublisher has taken technical and administrative measures to assure authenticity,security and accessibility.

According to intellectual property law the author has the right to bementioned when his/her work is accessed as described above and to be protectedagainst infringement.

For additional information about the Linköping University Electronic Pressand its procedures for publication and for assurance of document integrity,please refer to its WWW home page: http://www.ep.liu.se/

© David Lindh

Master of Science Thesis in Media Technology

Department of Science and Technology, Linköping University, 2019

The effect of Assessment

on Motivation in a Virtual

Reality based Serious

Game

David Lindh

Master of Science Thesis in Media Technology

The effect of Assessment on Motivation in a Virtual Reality based Serious

Game:

David Lindh

LiU-ITN-TEK-A-20/010--SE

Supervisor: Karljohan Lundin Palmerius

itn, Linköpings universitet

Saad Azhar

ABB Corporate Research

Examiner: Daniel Jönsson

itn, Linköpings universitet

Science and Technology

Department of Science and Technology

Linköping University

SE-601 74 Norrköping, Sweden

Copyright © 2019 David Lindh

Sammanfattning

Projektet började för att företaget ABB ville använda VR och gamification till att

göra ett program i utbildningssyfte, ett så kallat Serious Game. Programmet är en

prototyp ämnad att fungera som ett verktyg med syfte att bistå i utbildningen av

ny personal för företaget. Existerande forskning visar att gamification kan vara

ett kraftfullt verktyg som kan ha en signifikant effekt på inlärningen hos använ-

darna. Den nuvarande forskningen inom gamification riktar sig mot att etablera

en taxonomi av gamification attribut och mäta deras effekt på olika attribut av in-

lärning. Rapporten beskriver jobbet som gjorts för att utveckla en Serious Game

prototyp åt företaget. Den beskriver också studien som gjorts för att mäta effek-

ten av gamification attributet som kallas “Assessment” på inlärnings attributet

som kallas “Motivation”.

Det var på ABB’s begäran att prototypen utvecklas för Virtual Reality,

programmet som kallas Unity och API’t vid namn SteamVR användes vid ut-

vecklingen. Genom att använda den inledande forskningen på gamification och

erfarenhet samt undersökning av olika spel togs ett koncept för designen på pro-

totypen fram. På grund av de tidsbegränsningar som fanns kunde inte hela de-

sign konceptet implementeras och användas i studien. Assessment systemet som

utvecklades använde sig av vanliga element från liknande system som erfaren-

hetspoäng, bedrifter, och vanlig tidtagning för att utvärdera spelarens prestation.

Ytterligare information gavs till spelaren i textform om deras prestation tillsam-

mans med tips på förbättringar på spelarens prestation.

En sektion av studien användes för att mäta dem medverkandes upp-

fattade värde av programmet. Resultatet på denna mätning hade generellt höga

värden. Dessutom sa flera medverkande efter deltagande i studien att de gillade

prototypen och trodde den hade värde som ett säkert sätt att utbilda personal.

iii

iv Sammanfattning

Eftersom programmet i sig självt är ett resultat så presenteras det mer genom-

gående i kapitel 3 i den här rapporten. Resultatet av studien visade att ingen

statistiskt signifikant effekt uppmättes vilket tyder på att assessment systemet

inte har någon märkbar påverkan på motivationen hos användaren. Det finns fle-

ra faktorer som kan ha påverkat resultatet. De flesta influenser tas upp i detalj

senare i rapporten. Slutsatsen blir att effekten av ett assessment system kan på-

verka något annat attribut av lärandet men har i detta fall inte haft någon effekt

på motivationen. Med syfte att kunna återskapa resultatet har relevanta delar av

programmet och metoden som användes för studien beskrivits. Förhoppningsvis

kommer rapporten vara till hjälp för läsaren om denne skall utveckla ett eget

effektivt Serious Game med gamification attribut som assessment inkluderat.

Abstract

ABB desired to utilize VR and gamification to make a learning program, or Se-

rious Game, that would be a useful tool in educating new personnel. Existing

research indicates that using gamification can have a positive impact on the learn-

ing curve of the users. Currently the research into gamification aim to establish a

taxonomy of gamification attributes and measure their effect on various learning

outcomes. This report details the work that was done to develop a prototype Se-

rious Game for the company. It also details the study made on the effects of the

gamification attribute called Assessment on the learning outcome called Motiva-

tion.

It was at ABB’s request the prototype be developed for use in Virtual

Reality. For this purpose, a program called Unity was used and an API called

SteamVR. Using research made on gamification and its use in various other games

a concept design was made and later implemented. However, due to time con-

straints the whole concept could not be implemented for use in the study. The

assessment system developed used familiar attributes like experience, achieve-

ments and keeping time to evaluate the players performance. Furthermore, feed-

back was given on the players performance along with suggestions for improve-

ments to help the player improve.

A section of the study was used to measure how valuable the partici-

pants found the program, the results showed high values overall for the proto-

type. Additionally, many of the participants said they thought it’s a good way to

educate new personnel in a safe environment. Since the program itself is a result

it will be presented in greater detail in chapter 3 of this report. The results of the

study show that there was no statistically significant effect measured, which indi-

cates that the assessment system did not have any effect on motivation. There are

v

vi Abstract

various factors that may have affected the results. These influencing factors will

be discussed in detail later in the report. The conclusion are that although no sig-

nificant effect was measured the assessment system might have affected another

attribute. Furthermore, the way the program was made, and the study conducted

are described for the purpose of recreating the results. Hopefully, this report will

help inform the reader on how to create a more effective assessment system and

gamified Serious Game in the future.

Acknowledgments

A heartfelt thanks to my supervisors at ABB, Saad Azhar and Dawid Ziobro! They

have helped me a lot throughout this project and I am thankful for the time

I spent with them, you have been wonderful! Another round of thanks goes

to my supervisor at LIU, Karljohan Lundin Palmerius, and my examiner, Daniel

Jönsson,who both helped me with the report and when I was stuck on the project.

Västerås, Februari 2019

David Lindh

vii

Contents

Notation xi

1 Introduction 1

1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.2 Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.3 Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.4 Delimitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 Theory 7

2.1 Virtual Reality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.2 Serious Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.3 Gamification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3 The Implementation 13

3.1 Developing the Prototype . . . . . . . . . . . . . . . . . . . . . . . . 13

3.1.1 Design Process . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.1.2 Prototype Implementation . . . . . . . . . . . . . . . . . . . 17

3.2 Gamification Design . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

4 The Study 27

ix

x Contents

4.1 The IMI Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4.2 The Questionnaire . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

4.2.1 Qualitative part . . . . . . . . . . . . . . . . . . . . . . . . . 29

4.2.2 Quantitative part . . . . . . . . . . . . . . . . . . . . . . . . 30

4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

4.3.1 Quantitative study . . . . . . . . . . . . . . . . . . . . . . . 31

4.3.2 Qualitative study . . . . . . . . . . . . . . . . . . . . . . . . 33

5 Discussion 37

6 Conclusions 41

6.1 Answering the Questions . . . . . . . . . . . . . . . . . . . . . . . . 41

6.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

Bibliography 45

Appendix A 49

.1 Questionnaire . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

.2 Tables of Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

.2.1 Group A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

.2.2 Group B . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

Index 55

Notation

Common Abbreviations

Abbreviation Meaning

vr Virtual Reality

sg Serious Games

hmd Head Mounted Display

api Application Programming Interface, a computing in-

terface which defines interactions between multiple

software intermediaries

steamvr SteamVR is an api for developing games

ovr Oculus Rift’s VR api for developing games

viveport HTC Vive’s VR api for developing games

mmorpg Massively Multiplayer Online Role Playing Games

moba Multiplayer Online Battle Arena

xi

1Introduction

This master thesis report intends to clarify the methods and ideas behind the cre-

ation of a training program that was developed for a company called ABB. The

prototype that was developed is meant to provide ABB with a safe and easily ac-

cessible way of training new service engineers. In order to understand how to

make an effective learning tool some key concepts must be introduced. The type

of the program is called a “Serious Game” or “Applied Game”. A Serious Game

is defined as a game with a primary purpose other than entertainment. Serious

games inherently use some form of “Gamification” for it’s make up. Gamifica-

tion refers to the application of game design elements in non-game contexts such

as a learning program or training simulation. This program was made using a

software called Unity and is meant to be used in “Virtual Reality” (VR). Much

research shows that gamification can have a positive effect on learning for exam-

ple by [1, 4, 9, 12]. This is a desirable outcome in many areas of application

for Serious Games and is therefore the subject of much research. Virtual Reality

(VR for short) is a step up in the Gamification attribute called “Immersion” and

1

2 1 Introduction

is therefore considered to also have a positive impact on learning, Krokos et al.

[6]. To answer the research questions presented in this chapter a study was made

with the goal of investigating any effect the Gamification attribute “Assessment”

could have on the learning outcome called “Motivation”. Linking one attribute of

gamification to an attribute of learning this way is what the research community

now focuses on according to Bedwell et al. [1].

1.1 Background

The master thesis work was suggested to a division of the company called ABB

as a continuation of the summer work that was done for them. They needed a

prototype of a program, a proof of concept to be developed further later, on ser-

vicing some of their products components. Especially important for them is the

safety of their workers, since working with high voltage or powerful machines

can be very dangerous, still new service engineers need to be trained. By prac-

ticing these servicing tasks in a virtual environment, the students learning this

trade have a safe place to learn before moving on to the real thing.

In several papers such as in [1, 8, 10] it is concluded that the existing

research often indicates that gamification can have a positive effect on learning.

Therefore, it stands to reason that using gamification along with VR is likely have

the same effect. Bedwell et al. and Boyle et al. [1, 3] also concludes that future Se-

rious Games using gamification elements may need to tailor their design towards

the purpose of that SG. For this to be possible further research is required to map

the gamification attributes effect on various attributes of learning as suggested in

Bedwell et al. [1].

Training new service engineers currently requires sending them out on

1.2 Purpose 3

site, often by airplane. On site trainees are tasked with performing repairs on

real components with the aid of a supervisor. This method of training new ser-

vice engineers is costly for the company as well as inefficient and unsafe for the

trainees. Furthermore, these circumstances limit the training opportunities for

the new service engineers to when something breaks and what gets broken. In

order to solve these problems ABB decided to investigate the possibility of using

a VR based Serious Game to help with the training of new service engineers.

1.2 Purpose

One of the purposes of the thesis is to investigate how a single gamification at-

tribute affects a single learning outcome. This direction is where the research

community, according to Bedwell et al. [1], wants to go to assist in making more

effective serious games in the future. In this case the attribute was assessment

and the learning outcome called motivation was chosen. It is also the purpose to

contribute to the current research on using not only gamification but VR in the

context of creating serious games. As VR represents a step up in immersion of

a program this new medium influences the gamification attribute called immer-

sion.

ABB’s purpose with this prototype is mainly to solve some of the pre-

viously mentioned problems with educating service engineers. A VR based SG

provides a safe place to train on any task whenever needed, without relying on

a real component breaking down. Additionally, it could be a much cheaper solu-

tion for the company than sending trainees around the world and simultaneously

reduce their carbon footprint. The program is not meant to replace real experi-

ence but rather act as a tool in preparation for the real thing. Hopefully, it will

be a more effective way of teaching the trainees than the traditional methods of

4 1 Introduction

study.

1.3 Questions

Based on the purposes of and reasons for the work there are several questions

that were formulated which this thesis aims to answer.

• What effect will the implementation of an assessment system have on the

motivation of the user?

• What effect will the implementation of an assessment system have on the

user’s perceived value of the program?

• Which gamification attributes are the most important in this type of SG?

1.4 Delimitations

To meet the required time constraints several delimitations were made when the

work began. The first delimitation was to focus on the implementation of one

task and the necessary accompanying functionality. Furthermore, this implemen-

tation had to contain the necessary components needed to perform the study. A

supporting scene, unrelated to the primary purpose, but aimed at familiarizing

the users with the VR controllers and navigation was included in the program.

Within the program that was made, functionality of non-essential yet

interactable components was limited to handles to turn, but nothing significant

happened when these actions were performed other than generating an error in

the assessment system. Other unnecessary components were ignored and left to

1.4 Delimitations 5

be developed if time permitted. Due to the primary time constraint, only a single

task could be developed to a sufficiently accurate level of realism.

The second delimitation was the narrowing of the scope of the project,

it was necessary from a scientific and practical point of view. Therefore, the gam-

ification attributes investigated as well as the effect on learning outcomes was

limited to one each. The program had to be usable for both major VR brands and

therefore the programming API had to be limited to SteamVR.

2Theory

To make a highly effective learning program, or Serious Game, there are some

key concepts that are used in one form or another. The main concept used in this

paper is called Gamification and VR is used as a medium for this program. In this

chapter these three key concepts are explained along with how they fit together

and their relevance for this program.

2.1 Virtual Reality

Virtual Reality, or VR for short, is a type of multi-immersive simulated experi-

ence displayed on special googles, [15]. It has gained much attention in the last

decade and since it’s rise in popularity it has been developed much and new ap-

plications are still being discovered. Virtual Reality has come to be a new and

interesting medium in many areas such as games, education and training, Radi-

anti et al. [11]. It is an exciting new way to play games and educate people with a

level of immersion that was not possible before. At the same time, it engages the

7

8 2 Theory

players physically which helps preventing the health problems with stationary

alternatives, Adrian Hansson [4]. In education it can be a helpful tool to visual-

ize what is being taught so the teachers can better teach their subject. It has also

been proven to help with recollection of what student were taught as mentioned

by Krokos et al. [6]. In training it has proven a useful method of giving trainees

relevant experience of a practical task without risking valuable equipment that

break easily or cost much.

In every area of application VR has an additional benefit, it provides a

safe space for the users. Using VR, a soldier can practice a raid or defusing a

bomb without risk of injury. A doctor can practice a difficult surgery without

risk to any patients. And with VR this can all be done in much more realistic

environments than ever before; this makes it an attractive tool for education and

learning. These are some of the best reasons for why VR can be a very useful tool

that has a wide variety of uses.

2.2 Serious Games

With the many technological advancements over the past few decades the interest

in SG as a tool for learning has increased significantly. Using both VR and gamifi-

cation many reports such as Hayn & Schreier [5] and by Adrian Hansson [4] also

Zikas et al.[16], to name a few, show that you can make a powerful learning tool

from this with many different uses. The definition of a Serious games according

to [8, 11? ] is essentially “a game designed for a primary purpose other than pure

entertainment”. This definition was first coined by C. Abt in 1970, in his book he

wrote “We are concerned with serious games in the sense that these games have

an explicit and carefully thought-out educational purpose and are not intended

to be played primarily for amusement.”, Majuri et al. (p. 1) [10]. He used gam-

2.3 Gamification 9

ing approaches to improve educative and training programs among other things,

and within a few years’ games came out that could be considered among the first

Serious Games. Odyssey by Magnavox, The Oregon Trail and Lemonade Stand

are some examples of early Serious Games.

Serious Games were especially popular within the military and in 1981

a game called The Bradley Trainer was released which was a games designed

to train new recruits to operate a Bradley tank, Laamarti et al. [7]. Military

purposes have since been behind quite a few SG’s many of which have a purpose

of training soldier in one or another scenario. Although there are examples of

SG’s being used in other areas such as business and education the interest in SG

increased along with the interest for regular video games. In 2010 a market study

showed the worldwide SG market was worth some 1,5 billion euros and growing

at a nearly 100% per year according to Laamarti et al. [7].

With the growth of the market and development of computers the re-

search of the subject grew and today SG’s are acknowledged as a powerful tool

in many forms of education and training, Ryu et al. [2]. Today it is a big field

of research and study, and most of us has encountered it in one way or another

whether we know it or not. With the development of gamification and VR the

modern SG has a whole new dimension of educational content tangled with im-

pressive stories, environments and functionality that makes it both fun and edu-

cational.

2.3 Gamification

Gamification is defined according to [14] as the application of game-design ele-

ments and game principles in non-game contexts. In short it means you try to

10 2 Theory

discern what makes a game interesting and fun and apply it to something else, in

this case a learning environment. Looking at the massive amounts of people play-

ing MMORPG’s and MOBA’s and other types of games for fun there must be some

reason for why it’s captured so many peoples interest. In gamification we are try-

ing to identify these elements of regular computer games that make it so enticing

and apply them in another setting. There are several papers and other articles

that agree that gamification can have a relevant effect on learning, for example

Majuri et al. [10]. Most papers on applying gamification in Serious Game do so

in the context of using a normal computer and screen, it was assumed that the

same effect can be achieved when using a VR system. There was however no clear

method found for how to measure these attributes or how exactly to implement

them. Another assumption was that as immersion is a gamification attribute, and

VR has a higher level of immersion than for example regular computers, the effect

on learning could be improved by using VR, Krokos et al.[6].

The research done by Bedwell et al. and Majuri et al. [1, 10] concludes

that there is enough research about whether gamification can have an effect on

learning to warrant further and more detailed research. In their paper Bedwell

et al. [1] it’s suggested future researchers try to find a link between single gami-

fication attributes and learning outcomes. The Bedwell et al. [1] report also has

a good summary table of papers that has listed the number of papers read with

an investigation between a learning outcome and gamification attribute. In ta-

ble 3 (p. 745-746) in Bedwell et al. [1] we can see which gamification attributes

were linked by which reports to affect specific of learning outcomes. It was this

table, combined with the company interests, that became the foundation for the

decision to investigate a link between assessment and motivation.

The way our program was developed had to be relevant to the company

while having research value at the same time. In order to fulfill both demands the

2.3 Gamification 11

program had to contain several gamification attributes that might be relevant to

the company while the main research attribute had to be easy to isolate from the

others. Assessment was a convenient choice as it can be added and removed with-

out changing too much of the overall functionality of the program. After study-

ing table 3 (p. 745-746) in Bedwell et al. [1] we can see there are only 2 listed

articles that indicate a positive relationship between Assessment and Motivation.

For this reason, it was concluded that there was room for further investigation in

between these attributes and therein lies the contribution to the research.

3The Implementation

In many cases finding well developed and tested methods in this specific area can

be near impossible. There are not many methods on how to specifically develop

a learning program nor how to implement the functionality it needs to contain.

Therefore, much of it is improvised and research into methods had to be limited

due to the time constraint of the project. Instead the methods used were derived

from discussion and meetings and interviews with different people in or outside

the department.

3.1 Developing the Prototype

It is important to give an accurate description of what was done and how, as well

as the design behind the implementation. This is the only way that the results can

potentially be reproduced and thereby verified. The three scenes are described

using words and figures to give as accurate a description of the design and im-

plementation process as possible. Furthermore, the design of the gamification

13

14 3 The Implementation

elements present in the program has a vital part to play in reproducing the re-

sults and warrants further description. Some aspects of the assessment system as

planned in its complete stage could not be implemented due to time limitations

and are therefore not detailed in this section.

3.1.1 Design Process

The design process began with a workshop meeting, ideas and thoughts were

gathered on necessary considerations with regards to the customer, the user and

the company. The next step was designing a concept for the prototype and es-

tablish what could be done in the time allotted. A first draft of the overview of

the prototype was then made, it was based on the company’s demands and the

research made into gamification along with the ideas from the workshop. Some

gamification elements were considered unnecessary for the prototype and were

left out of the design process, these were Game Fiction and Human Interaction.

Game Fiction refers to the development of some sort of story around the task and

Human Interaction refers to the ability of players to communicate in some way.

In the figure 3.1 a complete overview of the assessment system as it is intended

in a completed state is show. There are some aspects of the whole design in ??

that could not be implemented in the time given. These parts were not important

for the study, but ABB wanted a full design map to be developed later.

Virtual Reality had already been selected as the display system and the

development program selected is called Unity. The next important aspect of the

design is the primary task in the prototype. To select an appropriate task an

expert was consulted, the expert suggested a few tasks from which one was cho-

sen, servicing the filter of a cooling system. Furthermore, the expert provided

a description of the task as it was presented in the preexisting teaching mate-

3.1 Developing the Prototype 15

Figure 3.1: The whole concept for the assessment system

rial. From this point on implementation and design occurred parallel to each

other. Weekly meetings with the department that requested the prototype were

arranged, and another two expert meetings were later held. The weekly meetings

kept the department updated on the progress of the implementation and allowed

them to influence the development. The department mostly requested changes in

the scene design but were updated on the changes in the structure and function-

ality of the task. Although, many suggestions were cosmetic or including basic

functionality it was extremely useful for the program. An especially good idea

suggested during these meetings was to add a bucket to the workshop scene. It

added to the realism of the scene as water would come pouring out of the filter

once unscrewed and in real life it wouldn’t be dumped on the floor.

Two other expert meetings were held on separate occasions for the ex-

perts to evaluate the task and generate suggestions at improvements. Three dif-


ferent experts were consulted, one at each meeting, as their job took them away

all over the world whoever was around when an expert was needed was asked to

help. This proved fortunate as it provided inputs and suggestion from experts

with varied experience and knowledge. The first expert corrected the turning di-

rection of some handles and clarified that turning a nut included in the program

was unnecessary as the filter was unscrewed from its setting. In the last expert

meeting the previous suggestions were verified and the expert thought the task

was realistic and contained the necessary steps. However, a very useful sugges-

tion was made in this meeting that a step in the task could be skipped entirely

due to a feature in the cooling system.

The result of the design is a task that consists of several steps that can be

performed in varying orders of which only a few are correct. Firstly, there is an

alternative of turning the pumps off, or not, this possible alteration governs the

acquisition of a badge. Secondly, comes turning off the inlet and outlet valves to

the filter. Following this step, the drain nut is unscrewed using a wrench and the

filter empties of water, this is where the bucket comes into play and the chance

to gain another badge. Once this step is complete the filter can be unscrewed

from its setting and the inside of the filter extracted for cleaning. The filter is put

into separate positions on the table and this concludes the task. In the “tutorial”

mode the user is guided by arrows and text messages on the TV explaining what

to do and how. In the alternative mode called “practice” all the guidance was

removed to let the player practice the task. This is the only part of the prototype

that yields an evaluation of the players performance via the assessment system.

3.1 Developing the Prototype 17

3.1.2 Prototype Implementation

As mentioned previously the assignment was to make a functioning prototype

of a Serious Game in Virtual Reality for a certain servicing task. In the work

that preceded the master thesis another prototype had been developed, but the

conclusion of that work was that there was much room for improvement given

time. For this reason, the work began with a fresh start with some alterations

based on the experiences gained from making the previous version. Foremost,

among these changes is switching from the Oculus Rift API to the SteamVR API,

this brings several advantages. The main benefit is that the new API supports

controllers, HMD’s and sensors from both Oculus Rift and HTC Vive and a few

more brands. Additionally, the SteamVR API comes with several pre-built func-

tionalities that is free to use in the development of new software. Using this API

and its functionalities the current version of the prototype was developed. The

most notable of the pre-built functionalities used is the teleportation system and

algorithms for turning things, grabbing objects and making them throwable.

Three scenes were made using this API, all these scenes used some of

the pre-built algorithms in their implementation. Additionally, there are a few

packs found for free in the Unity store that were used in the scenes. For exam-

ple, a pack contained a model for the TV, another for the lamps yet others for

tables, internal filter, tool and bucket. Models related to the cooling system and

the power station were provided by the company, by using components of these

models the necessary parts were made interactable. Among these parts were han-

dles, switches, the filter canister and its nut. The scenes were each built to be

a finite space with walls and floors but for the sake of an effect in the tutorial

scene the ceiling was left out. For teleportation system to work an invisible copy

of the floors with a SteamVR “teleport area” component attached to each floor. A


SteamVR “player” component was added in each scene, this component synchro-

nizes the controllers and HMD’s movement and functionality with the players

movements and actions.

With this basis the scenes were built using the components in the packs

and those provided by the company while the floors, walls and ceilings were

made using standard geometry included in Unity. The functionality and the se-

quence of events in the task were implemented using C# scripts. Furthermore,

a static “player” variable was implemented to enable future development of the

program. This variable’s static declaration means it’s not deleted when the scene

changes and thus enables a player’s progress to be saved outside of a single scene.

The intention is to provide a basis from which a multi-player system can be made

for which each players progress is saved, and their performance can be accessed

and reviewed. For this reason, the variable contains a list of players and func-

tions for adding, removing and accessing players as well as the components in

the “player” class and sub-classes.

The task was eventually implemented exactly as described in the design

section of this chapter, but some changes were made along the way. Following the

expert meetings and meetings with the department the suggested changes made

had to be implemented. The bucket along with an animation of water gushing

from the filter was one of the implementations made. Another time the removal

of a step in which a nut was turned was made, while on another occasion a way

to bypass a step was implemented. Many changes of the design of the general

scenes were made but details about those are not relevant for the report.

3.2 Gamification Design 19

3.2 Gamification Design

The classification of gamification attributes follows the classification as suggested

in Bedwell et al. [1]. They made experienced “gamers” use a card sorting tech-

nique to narrow the scope of the previously established taxonomy consisting of

18 attributes. Their taxonomy divides the previous number in half totaling in 9

game elements or attributes. Using this taxonomy helps minimize the attribute

overlap problem, which is when changing one attribute changes another attribute

also. It becomes a problem when investigating the effect, a single attribute has on

another attribute as the effect can no longer be said to come solely from changing

one attribute. When the selection of which gamification attribute was made table

3 (p. 745-746) in Bedwell et al. [1] was a part of why assessment and motiva-

tion was chosen. In table 3 (p. 745-746) in Bedwell et al. [1] reports have been

listed that show a relation between the attribute and that learning outcome. From

these tables, along with the company’s requirement of an Assessment system, the

attribute called Assessment was chosen, and the learning outcome called Motiva-

tion. The company wanted to see an assessment system in the program regardless

of the choice of attribute for the study. According to the table there seems to be

room for more research on the connection between assessment and motivation,

but the table does not cover all the reports out there. Since the program was

made in VR and very few papers investigate Assessment and motivation in a VR

setting this became another good reason for this choice of attributes.

Almost none of the gamification aspects implemented in the program

followed a predefined method, instead observations of other games were used

as the method. Most gamification attributes came intrinsically with the require-

ments on the program by the company. Some of the listed attributes such as

“Human Interaction” and “Game Fiction” were left out entirely of the program.


However, the human interaction attribute was a part of the planning in the theo-

retical final version of the game.

“Action Language” refers to hardware through which we communicate

with the program, for example a mouse and keyboard. In this case it’s the con-

trollers and HMD from whichever of the platforms is in use, meaning HTC Vive

or Oculus Rift. In both cases they come with their own unique set of controllers,

sensors and HMD’s. The company required the game work on both HTC Vive and

Oculus Rift therefore an API called SteamVR was used to develop the program.

SteamVR has the benefit of supporting both types of controllers by identifying

common functionality of buttons. This way you can check if the SteamVR ver-

sion of the button is pressed when programming functionality in the game. The

sensors tracking your movements and that of the controllers would also be a part

of the action language.

The “Assessment” attribute and its effects is the focus of the study and

the implementation of it matters the most out of all the attributes. For this imple-

mentation the method used was observations in other games and studying com-

mon characteristics. The aim was to see what types of implementations were the

most common and recreate them to make an effective assessment system. How-

ever, it was discovered that many assessment systems are more complex than the

time allowed for implementation. Furthermore, some aspects of an assessment

system could not be implemented due to the programs state of completion. As-

pects like leaderboards would serve no use unless the game allows for multiple

players. Similarly, the stars system can’t work unless players have the chance to

perform tasks in real life and is therefore not implemented. But some aspects

such as score, experience, and text-based feedback on the tasks containing possi-

ble improvements or describing the errors were there. These aspects were imple-

mented as they were a common part of other games and make up the baseline for

3.2 Gamification Design 21

future functionality.

The “Rules & Goals” attribute is inherently present in the program as

per the request of ABB and the nature of a SG. There must be some guidance

for the steps the player needs to take to complete the tasks. In the program the

chosen method for this guidance are for one arrow that point to the component

that is the current target. Additionally, a short text displayed on a TV screen in

the virtual world that told the user what to do and what they were doing made

this attribute. In the case of a task involving rotating an object a color-coded

arrow was used to indicate the direction of rotation and how much rotation was

left.

The attribute “Conflict & Challenge” is implemented in the form of a

linear change in difficulty. As there is only one task now this could be expanded

upon by adding more tasks of varying difficulty. However, the main change in

difficulty in the existing task occurs when going from tutorial mode to practice

mode in the workshop scene. In this step the guiding arrows and the instructions

disappear, and the player is tasked with performing the task without guidance.

“Control” is an attribute which Bedwell et al. [1] refers to as the amount

of control the user have over their surroundings. In the application this is limited

to choosing where to go and what to do along with interacting with objects which

have been made interactable. There is no functionality implemented beyond the

workshop and tutorial scene. Due to the SG angle, the game has little need of

allowing the player to control their environment to a greater extent.

The attribute “Environment” would always exist in any program in one

form or another that has an environment. The environment in the program is

mostly simple and limited so the user cannot get distracted and to make the

program run smoother.


“Immersion” as a gamification attribute refers to the extra sensory ef-

fects that makes a game seem real to the player. It can be realistic sound or

vibrations in the controllers that mark an important event or something else that

adds shock and intensity to the game. In the prototype this attribute presents

as sound of the pump and vibrations in controllers when interacting with some

objects. Naturally, the VR environment is a step up in immersion from the tradi-

tional computer screen.

3.3 Results

Figure 3.2: The view of the starting scene

The prototype program consists of three different scenes, a starting scene

followed by a tutorial scene and finally a workshop scene. The starting scene Fig-

ure 3.2 is the least extensively developed of the scenes but has a vital function.

This is where the player selects what task to work on or if they want to try the tu-

torial, when a task is ended by the player they are sent back to this scene. Further-

more, a vital function of this scene is to minimize the computational demands of

each scene thus making sure the program runs smoothly.

The tutorial scene, as seen in the figures 3.3a and 3.3b , is where the

player goes if they are new to VR. Here, the player can get familiarized with

3.3 Results 23

(a) Left side (b) Right side

Figure 3.3: Tutorial Scene

the controllers, the VR environment and in-game functionality. For this reason,

the tutorial scene contains interactable objects from the workshop scene. These

objects function the same as in the workshop scene so that the player can learn

how they function before encountering them there.

Figure 3.4: Image of the model in the workshop scene

The workshop scene Figure 3.4 is the scene that contains the task and all

the functionality related to this scene. The intention is that when the program is

developed further there will be many such scenes, all containing the models and

functionality relevant to that task. Currently, there is only one of these scenes

since the development of a single task has been the primary focus.


Figure 3.5: Image of the TV in the workshop scene

In the workshop scene the player will be asked to select what they want

to do via buttons on a TV screen in the scene Figure 3.5. The options on the

screen are either a guided tutorial of the task or practicing the task without any

guidance. In the guided tutorial the player will receive instructions in text form

via the TV screen and by arrows pointing to the relevant object in the model.

Figure 3.6: Turning the wrench in the quest

The idea is that the text tells you what you’re doing while the arrows

show you where the correct component is. Occasionally, the arrows have a dif-

ferent function, to show direction of rotation of an object, as seen in figure 3.6.

Additionally, in the rotation case the arrows are color coded and turns from red

3.3 Results 25

to green with a speed relative to the angle the object has been turned.

4The Study

To measure the effect of the Assessment system on Motivation a study was con-

ducted using the Intrinsic Motivation Inventory method. To measure this effect

in a scientifically correct way an established method for making such a measure-

ment is needed along with a good placement of when to make the measurements.

If all of this is done correctly a correlation between the assessment system and

the effect on motivation can be found and measured.

In order to show that the measured effect on motivation is only from a

single gamification attribute, that attribute needs to be isolated and the results

compared with a control group. For the study to be scientifically valid the pro-

grams must be similar in all ways except the implementations related to the

assessment attribute. The logical approach was that the study must have two

groups one to compare the results of the other with. One group would be shown

the assessment system and the other group would not, thus both groups encoun-

tered the same program except for the assessment system itself.

27

28 4 The Study

Furthermore, the people in each group should be of somewhat similar

educational level and background. It was decided that selecting the participants

from among the people working at ABB, within the same building where the

project was developed, would be prudent. When recruiting participants. the

method was therefore to go from door to door and ask people and whoever had

time was given a spot.

4.1 The IMI Method

To measure the effect on motivation an already established method was found

and used, it’s called the Intrinsic Motivation Inventory. It consists of subscales

with questions for measuring motivation as well as positive and negative predic-

tors on motivation. The IMI method [13] is a measurement device intended to

assess participants subjective experience of a certain activity. There are several

subscales in the method for measuring different things called Interest & Enjoy-

ment, Perceived Competence, Effort, Value & Usefulness, felt Pressure & Tension

and Perceived Choice. Commonly, only a few subscales are selected to make up a

study and it is up to the creator of the study to select what is relevant in their case.

The subscales called Enjoyment and Interest, Perceived Competence, Pressure

and Tension and Value and Usefulness were chosen to make up the study. The

Interest & Enjoyment subscale is considered the self-report measure of intrinsic

motivation and is the only subscale that measure intrinsic motivation. Perceived

Competence is theorized to be a positive predictor of self-report and behavioral

measures of intrinsic motivation. The Pressure & Tension is a negative predictor

of self-report and behavioral measures of intrinsic motivation. Both these predic-

tors were added to be thorough and in the hopes of finding some unknown but

surprising result that influenced motivation in some way. The Value and Useful-

4.2 The Questionnaire 29

ness subscale was added for the company and to answer a research question, the

idea was that this subscale would be useful for the company to see how useful

the participants found the program. The IMI method states that this subscale is

based in the idea that people internalize and become self-regulating with respect

to activities that they find useful or valuable for themselves.

4.2 The Questionnaire

The questionnaire was divided into two sections, one section filled with questions

following the IMI method and the other section contained qualitative questions.

Both these types of questions can be valuable in a study, especially if one method

of questioning provides no clear answer or relevant data. The questionnaire can

be found in the appendix of the report.

4.2.1 Qualitative part

The qualitative section followed no singular method other than general questions

about what the participants thought about some elements of the program. This

qualitative part was only given after they had finished the task so it would reflect

on the whole program rather than just the assessment system, it had no direct

link to motivation. The purpose was to see if participants would mention the as-

sessment system when asked about valuable feedback. It was also about finding

suggestions for improvements and to gather opinions on the program not covered

by the specific questions.

30 4 The Study

4.2.2 Quantitative part

The quantitative part contained three questions on each of the four subscales in-

cluded, one on intrinsic motivation, two predictors of intrinsic motivation and

one on usefulness. During the study the subjects were presented with a set of

different ordered and worded questions before and after performing the evalu-

ated task. The idea was to make a comparison of the answers before and after

performing the main task for both groups. If this resulted in any significant mea-

surable difference between the two groups, then conclusions can be drawn as

to the effect the assessment system. A Likert scale of 5 points was used for the

measurement of the subscales according to the IMI method as presented earlier.

Upon the study’s completion the collected values of the questions belonging to

each subscale were averaged. Then all averages were summarized again for the

whole group and divided by the number of participants in that group. These fi-

nal averages were used in the bar-chart and show us how much the average of

the different subscales changed between the two places of measurement. The par-

ticipants individual averages for each subscale can be used to make a statistical

analysis to test the significance of the measured difference.

4.3 Results

The results of the study are divided into the same two parts that make up the

study. In the quantitative study the results and math that goes with it are pre-

sented and in the qualitative part some suggestions and mistakes made by partic-

ipants are dealt with. Only the quantitative results are relevant to the research

questions but with that said the qualitative section has some merit and good sug-

gestions for future work as well.

4.3 Results 31

4.3.1 Quantitative study

There were two groups in the study, for future simplicity they will be referred to

as Group A or B, thus no further explanation of which is which will be needed.

Group A was shown the assessment system and group B was the control group

to whom the assessment system was hidden. Group A had a total of NA = 13

participants and group B had a total of NB = 14 participants. Each participant

responded with a value from 1 to 5 for each question, there were 3 questions

per subscale. The value per participant in each subscale is the mean of the three

questions belonging to each subscale. Due to the chosen number of questions per

subscale most results ended up being in the form of X/3. But in order to make

the summary easier the displayed results are not divided by 3 in the tables until

later. Instead the final division of the average, by the size of the group, is divided

by NA*3 or NB*3 instead. See appendix A for tables of the results.

Using statistical analysis, a confidence interval for each subscale in each

group for each set of answers could be calculated. The confidence interval was

calculated using a confidence level of 95%, this means there was a 95% chance

the correct value of the average for the experiment was within this confidence

interval. Using the confidence interval, it can be determined if two data sets

differ significantly enough that any conclusions can be drawn from them. This

is done by checking if the mean value ± confidence interval overlaps for the two

measures. If they do that means they could have a common mean and so the

difference of the means could be from a statistical anomaly.

The values can be observed in the tables in appendix A, when properly

compared we see that the confidence intervals will overlap in every case of the re-

sults in the study. This means that the data doesn’t indicate at a relation between

the assessment system and motivation, at best it can be said that while there may

32 4 The Study

Figure 4.1: Barchart of the subscale averages

be a very small effect it is so small it may just be a statistical anomaly. The cause

for this might be it’s the actual truth or the result of mistakes made along the way

in the program or the study. Some reasons and theories for why these measures

had this result will be discussed to a greater extent in the following chapters.

The result of this work are visualized in the bar chart as seen in figure

4.1. What it shows is the values measured for each subscale visualized, blue for

group A and brown for group B while darker shade is before practice and lighter

shade is after practice. Group A was shown the assessment system and group B

was the control group.

The darker colors represent the first measurement taken with the partic-

ipants during the study and the lighter colors are the second measurement taken

at the end of each participants session. Along the y-axis is the summarized aver-

ages from all the participants results and along the x-axis are the grouped results

4.3 Results 33

of each subscale. The abbreviation V/U means Value & Usefulness, I/E stands for

Interest & Enjoyment, P/T is for Pressure & Tension and PC refers to Perceived

Competence.

4.3.2 Qualitative study

As a part of the study there were a few qualitative questions with the purpose

of collecting a bit of feedback about improvements and what participants liked

or disliked. A few people had some initial problems with the teleportation, but

most of them were good with it after doing the tutorial scene and tutorial for

the task. Some mention problems detecting which way to turn the canister in

the game, a problem that had been brought up previously. Since the texture

of the canister is one color without a pattern and the canister itself is perfectly

symmetric it is hard, almost impossible, to see if it’s rotating. For this reason, an

arrow had been implemented that was meant to indicate the correct direction of

rotation. It changed color from red to green when turning the right way and was

pointing in the intended rotational direction, an example can be seen in Figure

3.6, however many seem to have forgotten this when they got to the practice part.

Some mentioned the vibrations as an indicator of an object being interactable,

which is not always the case. Many seem to not have noticed the vibrations at

all, possibly due to the battery in the controllers being low after a few days, and

some thought the vibrations were confusing. The intention of the vibrations was

to indicate that the player is grabbing something or doing an action like turning

a valve.

Predictably, many in Group B felt like they did not get any feedback

at all since the feedback was more subtle than the visual feedback Group A got.

Many Group B participants often asked what kind of feedback the questionnaire

34 4 The Study

was referring to. This was resolved by referring to the other feedback for example

the haptic in the controllers, most of them mentioned the feedback in the tutorial

instead. Another issue that came up was the necessity to look at the screen during

the tutorial. A few forgot to do this despite the audio indication that was added

to tell the user that the instructions on the screen had changed.

Among the users in Group A with the assessment system there were

some who thought the instructions for gaining the achievements were a bit un-

clear. There was, however, implemented hints in the details section that showed

up after completing the task, but it seems they were largely overlooked or not

reflected on by these participants. Many liked the time aspect of the assessment

system and mentioned wanting to beat their own time. However, one person men-

tioned that it might be beneficial to have an exploratory mode where the player

can just try doing the task without incentives to improve. Many participants

asked about the achievements and how to gain them after their participation was

over, but few tried during their participation. This could indicate that they found

it motivating at least to some degree.

Members of both groups reported problems grabbing a hold of an item

on occasion which is an issue previously known, the cause of which is sensor

placement. It happens when a person tries to grab something but is near the

edge or outside the sensor tracking range, the controller will then stop reacting

to the user’s movements correctly. It’s believed that if better, or more, sensors

were available this issue could be avoided entirely.

While the participants were immersed the supervisor took notes on their

behavior and a few common issues were observed. One of these was that quite

a few tended to reach over or avoid walking into items in the game as if they

were real. Sometimes this evasive behavior came to the point where the super-

4.3 Results 35

visor had to mention it to prevent an accident. Others had no problem going

straight through objects to get where they wanted to go or reach from a good an-

gle. Another common behavior that became a problem occurred when turning a

handle, the participants tended to do a turning motion about halfway, then con-

tinue with a straight motion along the tangent of the previous circular motion.

This caused some troubles turning the handles for a few of the participants, but

most resolved the issue by making another grab at the handle and completing

the turning in two steps. This problem was never the case with the turning of the

wrench however as all participants managed to turn it without fail, after going

through the tutorial.

5Discussion

Surprisingly many people showed an interest in the project and in VR which

made the effort put into making the program feel worthwhile and relevant. As

seen in the 4.1 the Value & Usefulness subscale had reasonably high values over-

all, a factor worth mentioning here is the novelty effect but there are many other

possibilities. Hopefully, this means that the participants found the program to

be useful and worth developing further in the future.

Before the work on the project began there was a period of introduc-

tory studies made in the subject of gamification and Serious Games. Despite

these studies no clear method on how precisely to measure the effect of a sin-

gle attribute was found. Therefore, the method described on how to measure

the results can only be said to follow a logical approach to this problem. This

homemade method for measuring the effect of one attribute was not tested but

was presented to the others in the team in the hopes of finding suggestions for

improvements. They were unsure about its validity in a scientific research con-

text but that the method presented seemed to make sense. In addition, when

37

38 5 Discussion

they were told that other papers don’t present any clear method for doing this,

they took it to mean it’s there is no clear method and in a big way it’s up to the

researcher. It’s possible that it was erroneous not spend more time looking for

some guidelines on how to make a measurement such as this. But since none had

been found after a few weeks of reading various research papers it was assumed

that the method of measuring was unique and adapted to each situation. The

method used in this paper was to measure twice, before and after encountering

the assessment system for both groups. As mentioned in “The Study” chapter par-

ticipants had to be given a chance to react to the feedback from the assessment

system. Therefore, all participants were tasked with performing both practice

and tutorial stages two times each. Although this made sense for Group A it was

confusing for Group B, but the iteration had to be kept so the group results were

comparable.

Because of the gap between measures and by including error feedback in

the assessment system some unintentional mistakes with attribute overlap might

have been made. The text feedback in the assessment system could be viewed as

a part of the Rules & Goals attribute. This is because in the feedback a text-area

tells you what errors you made and gives hints at ways you can improve. This

belongs in the Rules & Goals attribute domain and it was implemented without

considering this overlap. This is however an interesting feature for the company

but most importantly useful for the player. A possible solution to this would

have been to let the participants try the practice mode before making the first

measurement then activate the assessment system, or not, and make the second

measure. Another attribute overlap that might affect the study is in the measure-

ments. The first measurement is taken right before the switch from tutorial to

practice mode. This translates to a change in the Conflict & Challenge attribute

experienced by the player, who is now asked to do the task without guidance. The

39

idea was that by measuring at the same point in the control group, Group B, we

could see the difference between measures. The control group would experience

the same change in the Conflict & Challenge attribute and the results would be

comparable. This difference would then be from the assessment system, but the

difference was too small to say the cause isn’t a statistical anomaly. This could

mean a few things, either the assessment system had little to no effect on motiva-

tion or there was a flaw with the study or the implementation of the assessment

system. Alternatively, attribute overlap caused two different effects to cancel each

other out and there is currently no way of knowing which is true. Ideally, with

more time available for the study, the participants would be allowed an extra it-

eration of the practice part. If the first measure is taken on the second iteration

and the third iteration left to react to the assessment system this problem might

have been avoided.

Another factor to consider when discussing the result and the cause of

it is the IMI method [13] and its application in the study. An error was made

when making the study, specifically a section of the text describing the method

was overlooked. When the text was reread after the study was over it was discov-

ered that standard practice is to have maybe 60% of the questions in the relevant

subscale. In this case the Interest & Enjoyment, (IE), subscale is the only subscale

that measures intrinsic motivation. The predictor subscales are irrelevant to this

study except that they take behavioral measures of intrinsic motivation. To any-

one using this method for the same purpose as this study the recommendation is

to exclude the predictor subscales completely. Either way this was unknown at

the time and hopefully by reporting this none who reads this report will make

the same mistake. Additionally, in the questionnaire a 5-point Likert scale was

used for the questions, but it’s possible to use a 7-point Likert scale instead. The

increased numerical options allow for a more precise representation of the user’s

40 5 Discussion

opinion which could have affected the results.

In the implementation of the program and its functionality the biggest

issue is the incomplete state of the program. Several aspects of the program could

not be included due to time constraints which caused a few assessment system

components to be left out. An example of this is the stars system, a star was meant

to be an additional ranking of the players proficiency at the task. Stars were to

be granted after the player successfully completed a service task in real life of a

component. The purpose of this was to emphasize the value of real experience

and using the programs reward system to increase motivation for gaining real

life experience. Another excluded component was a chat or mail system and

leaderboards, meant to motivate player with a social and competitive angle to

the game. It’s possible that with a more comprehensive version of the program

the results of the study would have been different, but then so would the method

of the study itself. Additionally, there are some concerns that maybe the quality

of the design of the assessment system might have influenced the results. Maybe

if the assessment system scoring screen was more entertaining to watch, in a

graphical sense, it could have influenced the outcome somewhat. No complaints

were had about this aspect but to exclude its possibility to affect the outcome

would be negligent. In a similar fashion some implementation of the gamification

attribute “Game Fiction” could positively impact the learning effectiveness of the

program.

6Conclusions

Should the program be remade completely there are several insights and conclu-

sions that can still be of use in that scenario. The research questions are answered

as well as can be, and based on the methods used, results obtained and discus-

sions in previous chapters. It’s suggested that the observations presented in this

chapter be considered in future development to avoid the issues discovered dur-

ing this project.

6.1 Answering the Questions

What effect will this implementation of an assessment system have on the mo-

tivation of the user? Given the data obtained the results clearly show that no

conclusive link has been found between the assessment attribute and motivation.

But considering the program is not complete and there are many assessment com-

ponents left out we cannot say that a link does not exist. Add to this the attribute

overlaps discovered it makes the waters of this puddle even murkier. Clearly, this

41

42 6 Conclusions

is but one implementation of an assessment system and there are many possible

variations. Therefore, we can definitively say that the effect on motivation by this

assessment system in this program is nonexistent or very small. However, other

research papers found in Bedwell et al. [1] could be worth a look to see the results

of other implementations.

What effect will the implementation of an assessment system have on the users

perceived value of this program? The answer to this question is almost the

same as the first question, although the differences between measures were larger

in the case of this subscale. It was still not enough to conclude that there was a

clear link between the assessment system and this subscale. To remind the reader

the idea of this subscale is that people become self-regulating when they find a

task or experience useful to themselves. It should be mentioned that the average

of this subscale was very high, so it is safe to assume that the participants did

find the program valuable and useful.

Which gamification attributes are the most important in this type of SG? The

research that was read during this project all indicate that each gamification at-

tribute is important in different ways, to a specific outcome. As such the current

research is trying to link these attributes to different outcomes to make it sim-

pler to create an effective SG in the future and to improve the current ones. But

there are some attributes that are intrinsic to the nature of any SG and thus will

always be there in one form or another. For example, Action Language is the

means by which we maneuver around in the program. It’s doubtful there could

be a game without it, the same goes for Immersion, Environment and Rules &

goals. It is however recommended that all gamification attributes be considered

important for any SG being made. Since the research on this subject is ongoing

6.2 Future Work 43

and incomplete the best that can be said currently is to assume that gamification

can impact learning outcome. Therefore, it’s unwise to exclude some parts of this

key concept until there is more conclusive research on the subject.

6.2 Future Work

For the future development of this program we suggest too start working on a

few more tasks and work out some kinks with the current one. For example,

sometimes when reloading the starting scene from the workshop scene, via the

in-game button next to the TV screen, the image became warped. The cause of

this problem is unknown, but perhaps the player position and HMD are not being

properly aligned when reloading the old scene. On another note the studies that

were made showed that Game Fiction can have a very big influence on players,

so it might be a good idea to have a story that ties the tasks together. Other

than the changes already suggested in the discussion and the conclusions some

redesigning of the general interface is needed. Visually pleasing menus and more

appealing environment might be a good idea to make the program appealing and

immersive to the player. Figure 6.1 show some graphical drafts of what the score

screen, completion screen and a profile screen for showing the results could look

like.

Figure 6.1: Concepts for future assessment screens

44 6 Conclusions

The tutorial scene could be extended to include future functionality and

further developed to better familiarize the player with VR. Keep making new

scenes and load in the models needed to make the task this way there is less

chance a scene gets too big and causes lots of lag when navigating around in it. A

closer look at the gamification attributes and their meaning according to Bedwell

et al. [1] is in order to familiarize yourselves with the concepts as they used them.

Finally, one of the most important things to consider for future develop-

ment is using some form of the existing divide and conquer strategy with regards

to the scenes in the prototype. Having very large scenes is computationally heavy,

and the program will be susceptible to lag. It was found that the lag was reduced

greatly when transitioning from a big scene to a starting scene from which indi-

vidual scenes containing only one task each was loaded. Furthermore, it might be

a good idea to make more detailed models of components that are the focus of a

task. And some less detailed version of the surrounding machinery to minimize

the computational load each iteration of the update function.

Bibliography

[1] Wendy L. Bedwell, Davin Pavlas, Kyle Heyne, Elizabeth H. Lazzara, and

Eduardo Salas. Toward a taxonomy linking game attributes to learn-

ing: An empirical study. Simulation & Gaming, 43(6):729–760, 2012.

doi: 10.1177/1046878112439444. URL https://doi.org/10.1177/

1046878112439444.

[2] Elizabeth A. Boyle, Thomas Hainey, Thomas M. Connolly, Grant Gray, Jef-

frey Earp, Michela Ott, Theodore Lim, Manuel Ninaus, Claudia Ribeiro,

and João Pereira. An update to the systematic literature review of empiri-

cal evidence of the impacts and outcomes of computer games and serious

games. Computers and Education, 94:178 – 192, 2016. ISSN 0360-1315.

doi: https://doi.org/10.1016/j.compedu.2015.11.003. URL http://www.

sciencedirect.com/science/article/pii/S0360131515300750.

[3] Gerald Christians. The origins and future of gamification. University of

South Carolina, page 65, 2018. URL https://scholarcommons.sc.

edu/senior_theses/254.

[4] Adrian Hansson. Fitness and exercising in virtual reality. 2018. URL

45

46 Bibliography

http://lup.lub.lu.se/luur/download?func=downloadFile&

recordOId=8963255&fileOId=8963256. Student Paper.

[5] D. Hayn and G. Schreier. VR medical gamification for training and educa-

tion, 2017. URL http://ebooks.iospress.nl/publication/46465.

[6] Eric Krokos, Catherine Plaisant, and Amitabh Varshney. Virtual memory

palaces: immersion aids recall. Virtual Reality, 23(1):1–15, Mar 2019. ISSN

1434-9957. doi: 10.1007/s10055-018-0346-3. URL https://doi.org/

10.1007/s10055-018-0346-3.

[7] Fedwa Laamarti, Mohamad Eid, and Abdulmotaleb El Saddik. An

overview of serious games. International Journal of Computer

Games Technology, 2014, 10 2014. doi: 10.1155/2014/358152.

URL https://www.researchgate.net/publication/286573155_

An_Overview_of_Serious_Games.

[8] Richard N. Landers. Developing a theory of gamified learning: Linking

serious games and gamification of learning. Simulation & Gaming, 45(6):

752–768, 2014. doi: 10.1177/1046878114563660. URL https://doi.

org/10.1177/1046878114563660.

[9] Chung Kwan Lo and Khe Hew. A comparison of flipped learning with gam-

ification, traditional learning, and online independent study: The effects on

students’ mathematics achievement and cognitive engagement. Interactive

Learning Environments, 01 2018. doi: 10.1080/10494820.2018.1541910.

[10] Jenni Majuri, Jonna Koivisto, and Juho Hamari. Gamification of education

and learning: A review of empirical literature. In Proceedings of the 2nd

International GamiFIN Conference, GamiFIN 2018. CEUR-WS, 2018.

[11] Jaziar Radianti, Tim A. Majchrzak, Jennifer Fromm, and Isabell Wohlge-

nannt. A systematic review of immersive virtual reality applications

Bibliography 47

for higher education: Design elements, lessons learned, and research

agenda. Computers & Education, 147:103778, 2020. ISSN 0360-1315.

doi: https://doi.org/10.1016/j.compedu.2019.103778. URL http://www.

sciencedirect.com/science/article/pii/S0360131519303276.

[12] Jung-Hee Ryu, Jin-Woo Park, Francis Sahngun Nahm, Young-Tae Jeon, Ah-

Young Oh, Hak Jong Lee, Jin-Hee Kim, and Sung-Hee Han. The effect of

gamification through a virtual reality on preoperative anxiety in pediatric

patients undergoing general anesthesia: A prospective, randomized, and

controlled trial. Journal of Clinical Medicine, 7(9), 2018. doi: 10.3390/

jcm7090284. URL http://www.mdpi.com/2077-0383/7/9/284.

[13] Unknown. Intrinsic motivation inventory, 2019.

URL http://selfdeterminationtheory.org/

intrinsic-motivation-inventory/.

[14] Unknown. Gamification, 2019. URL https://en.wikipedia.org/

wiki/Gamification.

[15] Unknown. Virtual reality, 2019. URL https://en.wikipedia.org/

wiki/Virtual_reality.

[16] Paul Zikas, Vasileios Bachlitzanakis, Margarita Papaefthymiou, Steve

Kateros, Stylianos Georgiou, Nikos Lydatakis, and George Papagiannakis.

Mixed reality serious games and gamification for smart education. In Eu-

ropean Conference on Games Based Learning, page 805. Academic Confer-

ences International Limited, 2016.

Appendix A

.1 Questionnaire

49

Questionnaire Before starting Yes/No

1. Have you ever tried VR before? 2. Do you have any experience with servicing cooling systems?

For each question below please indicate how true the statement is for you.

Not true Somewhat true Very true

1 2 3 4 5

After Tutorial

1. I think i learned all i needed to perform the (virtual) task without guidance. (V/U) 2. I enjoyed doing this tutorial. (I/E) 3. I experienced nervosity whilst performing this task. (P/T) 4. I think i did pretty well at this tutorial. (PC) 5. This tutorial was interesting to do. (I/E) 6. I think that doing this tutorial is useful for aspiring Service Engineers. (V/U) 7. I was not relaxed when doing this. (P/T) 8. After doing this tutorial i feel really competent. (PC) 9. I think the tutorial is important to do because it can teach you much. (V/U) 10.This tutorial had all of my attention whilst i was doing it. (I/E) 11. I am satisfied with my performance in this task. (PC) 12. I was anxious while working on this task. (P/T)

For each question below please indicate how true the statement is for you.

Not true Somewhat true Very true

1 2 3 4 5

After Practice

1. I enjoyed using this program very much. (I/E) 2. I was very nervous while performing this task. (P/T) 3. I am satisfied with my performance on this task. (PC) 4. I think this task is very useful to Service Engineers. (V/U) 5. My full attention was on the task while i was immersed. (I/E) 6. I think i did pretty well at this task. (PC) 7. I think this program could help me perform these task in the real world. (V/U) 8. I felt very anxious while working on this task. (P/T) 9. I would describe doing this task as interesting. (I/E) 10. I had achieved all i wanted to do in this task when i was done. (PC) 11. I felt pressured while doing this task. (P/T) 12. I think this is important because it is easier to remember than just instructions.

(V/U) Qualitative questions Was the feedback from the program good? Elaborate briefly. ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ Did it tell you any new information? (Y/N) How did the feedback make you feel? Elaborate briefly. ______________________________________________________________________ ______________________________________________________________________

52 Appendix A

.2 Tables of Results

Here are the tables containing all of the initial values and the calculations made

to make the barchart and the arrive at the conclusions.

.2.1 Group A

Group A: The group that was shown the assessment system

Subscale Result Average= X/(N*3)V/U 13+13+12+11+13+10+13+14+15+15+14

+13+9 = 1654.230769

I/E 10+15+15+14+14+15+14+15+14+15+14+15+14 = 184

4.717948

P/T 3+6+7+7+6+7+6+8+3+3+10+3+3 = 72 1.846153PC 10+11+8+12+12+10+11+10+11+13+12

+13+13 = 1463.743589

Table 1: Measure point 1

Subscale Standard Deviation Variance Confidence IntervalV/U 0.5991446895 0.7756410256 0.3256913168I/E 0.44816816133 0.09090909091 0.2436225112P/T 0.7771670374 0.7771670374 0.4530039009PC 0.4935481168 0.4935481168 0.2682908827

Table 2: Calculated Results

Subscale Result Average=X/(N*3)V/U 13+14+12+11+15+9+14+14+15+15+14

+13+13 = 1724.410256

I/E 9+14+14+14+15+12+13+15+15+15+14+15+12 = 177

4.538461

P/T 3+6+6+8+3+5+5+3+3+3+4+3+4= 56 1.435897PC 7+13+11+10+12+14+12+15+15+15+13

+14+14 = 1654.230769


.2 Tables of Results 53


Table 4: Calculated Results

.2.2 Group B

Group B: The group that were not shown the assessment system

Subscale Result Average= X/(N*3)V/U 15+13+13+10+13+13+12+10+12+14+11

+15+10+13 = 1744.166667

I/E 15+15+13+13+15+14+13+11+15+14+11+15+13+14 = 191

4.547619

P/T 3+10+7+4+12+6+4+5+4+9+11+9+5+7= 96

2.285714

PC 13+6+11+9+11+12+12+10+12+10+9+12+8+11 = 146

3.47619



Table 6: Calculated results

54 Appendix A

Subscale Result Average=X/(N*3)V/U 15+14+13+13+15+14+14+10+12+15+10

+14+12+13 = 1844.380952

I/E 15+12+13+10+14+14+11+9+15+15+12+15+11+13 = 179

4.261904

P/T 3+8+5+3+10+6+3+3+3+9+9+6+3+8 =79

1.880952

PC 15+9+13+11+13+14+11+9+13+12+11+15+11+11 = 168

4



Table 8: Calculated results

Index

api

abbreviation, xi

vr

abbreviation, xi

hmd

abbreviation, xi

mmorpg

abbreviation, xi

moba

abbreviation, xi

ovr

abbreviation, xi

sg

abbreviation, xi

steamvr

abbreviation, xi

viveport

abbreviation, xi

55

the effect of assessment on motivation in a virtual ...1471397/fulltext01.pdf ·...

Documents