improving student engagement with educational materialdtakpuie/project/front-end/...improving...
TRANSCRIPT
HONOURS PROJECT LITERATURE SYNTHESIS
Improving student engagement with educational
material
Deon Takpuie
Supervised by: Professor Sonia Berman
DEPARTMENT OF COMPUTER SCIENCE
UNIVERISTY OF CAPE TOWN
2012
NRF FUNDED RESEARCH
The financial assistance of the National Research Foundation (NRF) towards this research is hereby
acknowledged. Opinions expressed and conclusions arrived at, are those of the author and are not
necessarily to be attributed to the NRF.
Category Min Max Chosen
1 Requirement Analysis and Design 0 20 20
2 Theoretical Analysis 0 25 0
3 Experiment Design and Execution 0 20 15
4 System Development and Implementation 0 15 5
5 Results, Findings and Conclusion 10 20 10
6 Aim Formulation and Background Work 10 15 10
7 Quality of Report Writing and Presentation 10 10
8 Adherence to Project Proposal and Quality of Deliverables 10 10
9 Overall General Project Evaluation 0 10 0
Total marks 80
1
Abstract The Vula wiki tool is under-utilized in the computer science department at UCT, and in some other
departments has been replaced by alternative wiki tools that are easier to use. Since the wiki can be a
valuable educational tool, it was decided thatgamificiation should be used to increase the usability of the
Vula wiki on mobile phones. This led to the development of a system for computer science undergraduate
students which used an iterative user-centered design approach; consisting of a design, implementation
and evaluation of a prototype in each stage. Initially, two low-fidelity then two high-fidelity prototypes
are developed whilst incorporating user feedback from the previous iteration. At the same time,
gamification rules, which are influenced largely by the GameFlow criteria for player enjoyment in games,
are refined continually.
Thereafter, the final system is completed and evaluated with a summative evaluation using the
experimental process (on just 10 users). Overall, there has been no (significant) improvement in the
usability of the Vula wiki through gamification, but was statistically significantly faster for 1(out of 5)
tasks and had statistically significantly less errors for 1(out of 5) tasks.
However, this is the first time gamification has been introduced to the Vula wiki and has been met by
positive user feedback (although honesty is not assured).
We believe this project to be a first step in tackling an important problem because it seems not many
students do use the Vula wiki based on this research. Future extensions include improvement of game
rules and general wiki functionality of the system.
2
Acknowledgements
Firstly, I would like to thank my main supervisor, Professor Sonia Berman, who helped every step of the
way and provided valuable feedback on my writing. Also, I am grateful for the idea of making the wiki
more usable for students.
I would also like to thank the co-supervisor, Mr. Stephen Marquad, for always willing to answer
questions about the Vula system, of which he is the coordinator.
I would also like to thank my project partner, Reitumetse Chaka, who suggested the idea of gamification,
which is essential to our project solution. Furthermore, I hereby acknowledge his wonderful work ethic
and timeliness in meeting deliverables.
I would also like to thank Samsung for sponsoring me with a mobile phone for the duration of the project.
I hereby acknowledged all the students who participated in user testing for this project, I am most grateful
for the time (and even extra) that you have given me.
I would also like to thank my mother for her endless sacrifices and support, as well as my late father for
instilling a passion for technology in me. Likewise, I also hereby acknowledge my High School computer
science teacher, Mr. Dale Mckenzie, for introducing programming to me and his constant encouragement.
I would also to thank my family and friends for their support as well.
Last but not least, I thank Jesus Christ, my Lord and Saviour, as I could not have done this without Him.
He has given me strength, passion and much needed perseverance to finish this project.
3
Table of Contents Abstract ................................................................................................................................................... 1
Acknowledgements ................................................................................................................................. 2
List of Figures ......................................................................................................................................... 8
List of Tables .......................................................................................................................................... 8
1 Introduction ................................................................................................................................... 10
1.1 Problem Statement ................................................................................................................. 11
1.2 Research Questions ................................................................................................................ 11
1.3 Research Design..................................................................................................................... 12
1.4 Theoretical Framework .......................................................................................................... 12
1.5 System Overview ................................................................................................................... 12
1.6 Ethical Issues ......................................................................................................................... 13
1.7 Outline ................................................................................................................................... 13
2 Previous Work ............................................................................................................................... 15
2.1 Introduction ........................................................................................................................... 15
2.2 Gamification .......................................................................................................................... 15
2.2.1 Introduction .................................................................................................................... 15
2.2.2 Essential Game Elements ................................................................................................ 17
2.2.3 Benefits .......................................................................................................................... 17
2.2.4 Limitations ..................................................................................................................... 18
2.2.5 Psychology ..................................................................................................................... 18
2.2.6 Conclusion ..................................................................................................................... 18
2.3 Gamification methods at university ........................................................................................ 18
2.3.1 Introduction .................................................................................................................... 18
2.3.2 Types ............................................................................................................................. 19
2.3.3 Chosen gamification tool ................................................................................................ 21
2.3.4 Conclusion ..................................................................................................................... 22
2.4 Wiki ....................................................................................................................................... 22
2.4.1 Introduction .................................................................................................................... 22
2.4.2 Structure ......................................................................................................................... 22
2.4.3 Conclusion ..................................................................................................................... 22
2.5 Gamification of the wiki ......................................................................................................... 23
2.5.1 Introduction .................................................................................................................... 23
4
2.5.2 Gamifying the wiki ......................................................................................................... 23
2.5.3 Conclusion ..................................................................................................................... 23
2.6 Mobile Usability .................................................................................................................... 24
2.6.1 Introduction .................................................................................................................... 24
2.6.2 Techniques ..................................................................................................................... 24
2.6.3 Wiki UI Techniques ....................................................................................................... 26
2.6.4 User Evaluations ............................................................................................................ 27
2.6.5 Conclusion ..................................................................................................................... 28
2.7 Critical Comparison ............................................................................................................... 28
2.8 Gap in the knowledge ............................................................................................................. 29
2.9 Conclusion ............................................................................................................................. 29
3 Design Chapter .............................................................................................................................. 31
3.1 Introduction ........................................................................................................................... 31
3.2 Development platform ............................................................................................................ 31
3.3 Scope ..................................................................................................................................... 32
3.4 Methodology .......................................................................................................................... 32
3.4.1 Clients/Supervisors ......................................................................................................... 33
3.4.2 Literature Based ............................................................................................................. 34
3.4.3 Conceptual Model Extraction ......................................................................................... 35
3.5 Gamification Requirements .................................................................................................... 36
3.6 Prototypes .............................................................................................................................. 36
3.7 Evaluations ............................................................................................................................ 37
3.8 Conclusion ............................................................................................................................. 39
4 Low Fidelity Prototype Iterations ................................................................................................... 40
4.1 Introduction ........................................................................................................................... 40
4.2 Feasibility Study .................................................................................................................... 40
4.3 Paper Prototype1 .................................................................................................................... 40
4.3.1 Requirements Gathering ................................................................................................. 40
4.3.2 Design ............................................................................................................................ 41
4.3.3 Evaluation ...................................................................................................................... 42
4.3.4 Findings ......................................................................................................................... 43
4.4 Conclusions ........................................................................................................................... 43
5 High Fidelity Prototype Iterations .................................................................................................. 44
5
5.1 Introduction ........................................................................................................................... 44
5.2 Paper Prototype 2 ................................................................................................................... 44
5.2.1 Requirements Changes ................................................................................................... 44
5.2.2 Design ............................................................................................................................ 44
5.2.3 Evaluation ...................................................................................................................... 46
5.2.4 Findings ......................................................................................................................... 46
5.3 Horizontal Software Prototype ............................................................................................... 49
5.3.1 Design ............................................................................................................................ 49
5.4 Use Cases .............................................................................................................................. 49
5.4.1 Implementation .............................................................................................................. 50
5.4.2 Screenshots .................................................................................................................... 51
5.5 Evaluation and Findings ......................................................................................................... 52
5.5.1 Test Subjects .................................................................................................................. 52
5.5.2 Permission and consent................................................................................................... 52
5.5.3 Methodology .................................................................................................................. 52
5.5.4 Structure of evaluation .................................................................................................... 53
5.5.5 Findings ......................................................................................................................... 54
5.5.6 Changes recommended ................................................................................................... 57
5.6 Conclusion ............................................................................................................................. 60
6 Game Rules ................................................................................................................................... 61
6.1 Introduction ........................................................................................................................... 61
6.2 Final Game Rules ................................................................................................................... 61
6.3 Avatars .................................................................................................................................. 62
6.4 Badges ................................................................................................................................... 63
6.5 Rules not used ........................................................................................................................ 63
6.6 Conclusion ............................................................................................................................. 63
7 Final System Design and Implementation ...................................................................................... 65
7.1 Introduction ........................................................................................................................... 65
7.2 Feasibility Study .................................................................................................................... 65
7.2.1 Design ............................................................................................................................ 65
7.2.2 Implementation .............................................................................................................. 66
7.2.3 Evaluation ...................................................................................................................... 66
7.2.4 Feedback ........................................................................................................................ 66
6
7.3 Final Design ........................................................................................................................... 67
7.3.1 System Architecture ....................................................................................................... 67
7.3.2 DB-Schema .................................................................................................................... 68
7.4 Implementation ...................................................................................................................... 70
7.4.1 Methodology .................................................................................................................. 70
7.4.2 Tools and Platform ......................................................................................................... 71
7.4.3 Screenshots .................................................................................................................... 71
7.5 System Changes ..................................................................................................................... 73
7.5.1 Suggested changes not implemented ............................................................................... 73
7.5.2 Suggested changes implemented ..................................................................................... 73
7.5.3 General system changes made ........................................................................................ 74
7.6 Conclusion ............................................................................................................................. 74
8 Experimentation Design and Results .............................................................................................. 75
8.1 Introduction ........................................................................................................................... 75
8.2 Method .................................................................................................................................. 75
8.2.1 Hypotheses ..................................................................................................................... 75
8.2.2 Users chosen .................................................................................................................. 76
8.2.3 Experimental Design ...................................................................................................... 77
8.2.4 Control procedures ......................................................................................................... 78
8.2.5 Tasks .............................................................................................................................. 78
8.3 Findings and Results .............................................................................................................. 79
8.3.1 Results analysis and discussion ....................................................................................... 80
8.3.2 Usability Feedback ......................................................................................................... 83
8.3.3 Problems with the application ......................................................................................... 85
8.3.4 Problems beyond system scope ....................................................................................... 86
9 Conclusion .................................................................................................................................... 87
9.1 Future work............................................................................................................................ 88
10 References ................................................................................................................................. 89
11 Appendices ................................................................................................................................ 98
11.1 Paper Prototype 1 ................................................................................................................... 98
11.1.1 Screens ........................................................................................................................... 98
11.2 Paper Prototype 2 ................................................................................................................... 99
11.2.1 Screens ........................................................................................................................... 99
7
11.3 Windows Navigation Diagram of the Gamified Wiki ............................................................ 103
11.4 Description of Evaluation Tasks ........................................................................................... 104
11.5 The Final Evaluation Grid .................................................................................................... 107
11.6 Interview Questions ............................................................................................................. 107
11.7 Image Icons Used ................................................................................................................. 108
11.8 Statistical Data ..................................................................................................................... 110
8
List of Figures Figure 1: System Overview ................................................................................................................... 13
Figure 2:The Gamification Loop (Liu et al. 2011) .................................................................................. 16
Figure 3: EcoLand Picture(Shiraishi et al. 2009) .................................................................................... 17
Figure 4: Orientation app Screenshot (Fitz-walter, Tjondronegoro & Wyeth 2011a) ............................... 19
Figure 5: Student Improved Pass Rates (D. Bustard, M. Black, et al. 2011) ............................................ 20
Figure 6: Comparison of Gamification Techniques(Donovan 2012) ....................................................... 21
Figure 7: Microsoft Word file menu (Jones & Marsden 2005, p.230) ..................................................... 24
Figure 8: Tree Maps (Shneiderman 1992a) ............................................................................................ 27
Figure 9: User-centered design. (Telono 2012) ...................................................................................... 33
Figure 10: First Requirements Mind map ............................................................................................... 34
Figure 11: Sample GEL (game-enhanced learning) performance gauge tool (Bustard et al. 2011) .......... 35
Figure 12: Paper Sketch of Badges Screen ............................................................................................. 37
Figure 13: Home Screen–Paper.............................................................................................................. 41
Figure 14: Create Page Screen – Paper................................................................................................... 41
Figure 15: Edit Page Screen – Paper ...................................................................................................... 42
Figure 16: Badges Screen - Paper .......................................................................................................... 42
Figure 17: Home Screen - Paper Prototype 2.......................................................................................... 44
Figure 18: Create Page Screen - Paper ................................................................................................... 45
Figure 19: Edit Page Screen- Paper ........................................................................................................ 45
Figure 20: Badges Page Screen – Paper ................................................................................................. 46
Figure 21: Create Page Tutorial Screen .................................................................................................. 47
Figure 22: Gamified wiki Use Case Diagram ......................................................................................... 50
Figure 23: Horizontal Prototype Screens ................................................................................................ 52
Figure 24: Points gauge( the jQuery Foundation 2012a) redesigned by user ........................................... 58
Figure 25: New Created Page redesigned by user ................................................................................... 59
Figure 26: Feasibility Software Log-In .................................................................................................. 65
Figure 27: System Overview.................................................................................................................. 67
Figure 28: Database Schema .................................................................................................................. 69
Figure 29: Final Create Page Scenario ................................................................................................... 71
Figure 30: Final Gamification Screens ................................................................................................... 72
Figure 31: Average Clicks per Task graph ............................................................................................. 80
Figure 32: Average Task Completion Time graph .................................................................................. 81
Figure 33: Average Errors per Task graph ............................................................................................. 82
Figure 34: Windows Navigation Diagram - Gamified Wiki .................................................................. 104
List of Tables Table 1: Gamificiation - Benefits versus Limitations.............................................................................. 28
Table 2: Gamification Methods at University ......................................................................................... 28
Table 3: User Evaluations ...................................................................................................................... 29
Table 4: Evaluation Plan ........................................................................................................................ 53
Table 5: Navigation Results – Current Wiki ........................................................................................... 55
Table 6: Navigation Results- Gamified Wiki.......................................................................................... 55
9
Table 7: Achievements List ................................................................................................................... 62
Table 8: Avatar list ................................................................................................................................ 62
Table 9: Badges list ............................................................................................................................... 63
Table 10: Dependent and independent variables ..................................................................................... 75
Table 11: Final Evaluation Grid ........................................................................................................... 107
Table 12: Image Icons Used ................................................................................................................ 108
Table 13: Average Clicks Stats ............................................................................................................ 110
Table 14: Average Task Completion Time ........................................................................................... 111
Table 15: Average Errors per Task ...................................................................................................... 112
10
1 Introduction Wikipedia(Wikipedia 2012b), a free encyclopedia, is increasingly being used to engage with students
academically as can be seen from the 36% of students who used it in America in 2007(Rainie& Tancer
2007). This project attempts to use gamification to make the Vula wiki more usable on mobile phones.
Vula is a University of Cape Town(UCT) derivative of Sakai(S. Foundation 2012), an open source
learning management system . Gamification is the use of game techniques in non-game contexts (like a
wiki, in this case). Currently, the Vula wiki tool is under-utilized in the computer science department at
UCT, and in some other departments has been replaced by alternative wiki tools that are easier to
use(Berman 2012).
Therefore, the objective is to is to determine whether gamification improves the usability of the Vula wiki
on mobile phones, mainly in terms of key usability components speed and accuracy(Nigel Bevan 1995).
Indeed, the primary contribution of the research is to make the Vula wiki easier for students to use (on
mobile) although the addition of gamification to the wiki has never been done before either. Before
moving onto the research problem, it is necessary to explain key concepts used such as: gamification and
mobile usability.
Gamification
This originates with human computer interaction heuristics techniques dating back to the 1980s (Malone
1982) with experiments on how addition of elements inherited from game designs can enhance the
interest of an audience. Effects of gamification on university students has been investigated in a research
(Fitz-walter et al. 2011) and showed an improvement in interactions with the system information. There is
also a study done at UCT that attempted to determine what gamification techniques students find most
enjoyable (Donovan 2012).
An example of gamification in the education environment is the rewarding of students with incentives
(like bonus marks) to encourage participation (Bustard et al. 2007). In the context of the Vula system,
students can be awarded for their participation in wiki tools. The wiki has been chosen because all user
reads and writes are logged. The wiki tools will not immediately replace the current ones but will be part
of a new Vula course site that will test the implementation on a group of students. Furthermore, since the
wiki is considered on the mobile platform only, it is important to understand usability in mobile devices.
Mobile Usability
Usability, in the broader sense, is the ease of use of an application(N Bevan & Azuma 1997). Software
designed on mobile phones is quite different to that designed on desktops , in that mobile phones have
limited memory(Jong et al. 2008) , screen size, limited page types and more(Jones & Marsden 2005,
p.251). Therefore, careful consideration needs to be taken in the area of mobile usability whilst solving
the pressing issue of this project, to be discussed next.
11
1.1 Problem Statement The Vula wiki tool is under-utilized in the computer science department at UCT, and in some other
departments it has been replaced by alternative wiki tools that are easier to use(Berman 2012). Since it is
a potentially a valuable educational tool, and its short-snippet engagement style is particularly conducive
to mobile interaction, this project is designed to investigate the problem of incentivizing mobile usage of
the Vula wiki by means of gamification. This is particularly important given the poor throughput rates in
tertiary education(McMillan 2007) and the encouraging successes of gamified learning in other contexts
such as the improved pass rates (in first and second year computer science course) at the University of
Ulster via a gamified education tool (Bustard et al. 2011). However, since the front-end of the project
involves software engineering, it is also important to state the shareholders and their needs, before
moving to the research questions:
Clients The clients are Professor Sonia Berman, computer science HOD (head of department) at UCT and also Mr. Stephen Marquard, co-coordinator of Vula UCT. Their interests are to see the wiki made easier for
students to use on a mobile.
Users Are university of Cape Town (UCT) computer science (undergraduate) students and their objective is to
enjoy using the wiki tool.
1.2 Research Questions In order explore the above problem; this research investigates whether a gamifiedVula wiki on mobile is
more usable than the current Vula wiki on mobile. To justify, given that usability can be thought of as
ease of use of an application(N Bevan & Azuma 1997), the research question ties in with the project aim
of making the Vula wiki easier to use on mobile phones. Usability in this context is measured by
effectiveness/”errors” (average errors per task) and efficiency/”speed” (average clicks/time to complete a
task) and to a lesser extent user satisfaction(Nigel Bevan 1995). Therefore, the research question can be
broken down into three separate hypotheses (where mobile abbreviates mobile phone):
Hypotheses
1. A gamifiedVula wiki on mobile is more usable than the current Vula wiki on mobile.
1.1 For all tasks, agamifiedVula wiki on mobile requires fewer key presses to navigate than the
current Vula wiki on mobile.
1.2 For all tasks, agamifiedVula wiki on mobile requires less time to navigate than the current Vula
wiki on mobile.
1.3 For all tasks, agamifiedVula wiki on mobile is less error prone than the current Vula wiki on
mobile.
Of these, the dependent variable is system choice (either the Vula or gamified mobile wiki) and
independent variables are: number of clicks (measured relatively as screen transitions), task completion
time (measured absolutely in seconds) and number of errors (in completing tasks). Also, key success
factors for the experiment thus are to improve the Vula wiki in terms of number of clicks, task completion
12
time and errors as well as to critical analyze why such an improvement was or was not possible.
Therefore, an experiment (which is described next) is designed to evaluate this.
1.3 Research Design The research questions are to be evaluated in a summative evaluation (Nielsen 1993)using experimental
evaluation(Hezel Associates 2010)but first the participants need to be defined. The participants will be
computer science undergraduate students who are to be selected randomly. In the final evaluation, half the
participants will be novices (users who have not used either Vula or gamified wiki before) and the other
half experts (those who have used both wikis before). Also, instrumentation (or data collection tools) will
be an evaluation grid/coding sheet(Jones & Marsden 2005, p.202) which will record: the number of
clicks, task completion time and number of errors for every task. Furthermore, a post-experimentation
interview will also be used to extract users subjective system opinions(Jones & Marsden 2005, p.240). In
terms of evaluation procedure, the participants would test both systems for approximately 30 minutes
(whilst data is noted on evaluation grid) and thereafter complete a brief 10 minute post-experiment
interview.
1.4 Theoretical Framework A user-centered design framework(Marsden et al. 2008) is used to develop prototype iteratively and using
appropriate usability evaluations at each stage including: conceptual model extractions, heuristic
evaluations, formative evaluations and summative evaluation (for the final system). Also, GameFlow
criteria for player enjoyment in games(Sweetser& Wyeth 2005) has been applied to develop the
gamification rules.
1.5 System Overview In order to evaluate the research questions, a number of elements had to be designed and these are
illustrated with their allocation to the relevant group members (figure 1). Essentially, the system was
developed using a two-tier architecture consisting of a front-end (focus of this research) and a back-end.
This architecture was used for simplicity and allowed a modular development. To elaborate, the front-end
interacts with the back-end at a software/application level whereas the game rules alter it at a user
interface level (such as the inclusion of badges on a page). Likewise, the back-end interacts with Sakai
(for authentication and such) whilst providing the implementation of the game rules. In terms of
languages, the front-end was developed using jQuery Mobile framework( the jQuery Foundation 2012c)
for cross-platform development by using HTML, Ajax and JavaScript(with jQuery) to interact with the
back-end. In addition, there was only one device used for testing as this was the Samsung galaxy
ace(sponsored by the company) but more smartphones can be used in the future, as jQuery Mobile is
cross-platform(Godwin-Jones 2011) although from experience minor user interface tweaking is required
for some platforms(like Blackberry). Given these system components, the research questions can be
tested via experimental evaluation.
13
Figure 1: System Overview
1.6 Ethical Issues Firstly, the appropriate ethical clearance from the UCT Science Faculty Ethics clearance and access to
UCT students (from UCT student affairs department) has been obtained, as this was necessary to conduct
experiments. Furthermore, participants signed a consent form, were treated with respect, paid R30 as well
as thanked for their participation.
In terms of professional and intellectual property issues regarding the software development, a number
had to be taken into account. Firstly, commercial development environments such as
Dreamweaver(Adobe 2012) could not be used due to cost, so open-source alternatives such as notepad++
(Ho 2012) were used. Also, the jQuery Mobile platform( the jQuery Foundation 2012c) used to create the
user interface posed no legal constraints as it is created by the jQuery Foundation( the jQuery Foundation
2012b) , which is a non-profit trade association. Care had to be taken to use only free images online and
to download images from their original source not just from Google Images(Google 2012), for instance.
Furthermore, content in pages is allowed to be liked using a open source thumbs up icon(Clker.com
2012)as the familiar Facebook‟s (Facebook 2012) like button is copyrighted.
1.7 Outline Chapter 2 examines the foundation literature needed for the project, from the areas of gamification and
mobile usability. Chapter 3 provides an overview of the prototyping iterations of the project as well as
their evaluations. Chapters 4 and 5, expounds on these various low and high-fidelity prototypes as well as
Front-
End/Mobile
User
Interface
Back-End
Sakai
Game
Rules
Embedded
post
(request)
function
calls
Reitumetse Chaka Vula Deon Takpuie
14
their evaluations.Chapter 6 discusses the game rules of the system and how they are derived. Chapter 7
discusses the final implementation, in terms of its improvements from the final prototype. Chapter 8
evaluates the final system with a summative evaluation (through experimental process) and analyzes the
findings. Lastly, the final chapter concludes on these findings based on the hypotheses and also draws out
future extensions to the project.
15
2 Previous Work
2.1 Introduction There is much literature on making students engage more with educational material(Fischer & Troendle
2003). In particular, this paper focuses on the recent work of using online learning tools to increase
student engagement with academic material with the intention of improving their pass rates. The research
conducted pointed to a pioneering field, “gamification”, as being helpful in increasing student
engagement with educational material(Fitz-walter, Tjondronegoro & Wyeth 2011a). Hence, gamification
is evaluated broadly initially and then its particular implementations are compared. In fact, gamification is
the only method evaluated since it is often successful and simple to use(Burke & Hiltbrand 2011). In
between, all relevant aspects are defined appropriately such as the chosen tool of gamification (the wiki
tool). Lastly, the issues of usability and user testing on a mobile application are expounded on.
2.2 Gamification
2.2.1 Introduction
Origin
Following the success of a location-based service Foursquare(Crowley & Selvadurai 2009), the idea of
introducing gaming elements in non-game situation is now being discussed in health, education, media
and many more fields. Gamification has predecessors and similarities with the HCI and the games studies
fields. The term gamification was first documented in 2008 but only received widespread usage in the
second half of 2010(Deterding, Dixon, Khaled & Lennart 2011a).
Defintion
Gamification can be defined as the concept of introducing gamimg elements to non-gaming activities.
Authors sometimes use the term “gamify”, which is the application of gamification(Muntean 2002).
Games comprise of rules and competition towards outcomes or goals by human participants. Games
typically give rise to play but play is more general than games. Serious games use complete games for
non-entertainment purposes whereas gamified applications use game elements that do not lead to
complete games but add enjoyment to non-game activities(Deterding, Khaled, Nacke & Dixon 2011d).
To follow is a good overview of gamification for non-experts, showing the key interface level
components:
16
Figure 2:The Gamification Loop (Liu et al. 2011)
The diagram (figure 2) can be interpreted as a challenge leads to a win condition and so forth. Badges
represent achievement level of user and the point system is impacted by every other component.
Furthermore, gamification is also described as the process of adding game elements to an application in
order to enhance user experience(Fitz-walter, Tjondronegoro & Wyeth 2011a). However, there is still
some dispute in the video game and digital media industry when it comes to the various interpretations of
gamification thus some use different terms to describe the phenomenon such as “gameful design”
(Deterding, Dixon, Khaled & Lennart 2011a).
Examples
$25 billion was spent by Americans on video games in 2010, showing how natural games are to society.
Consequently, games/game elements could be used to increase engagement of users with e-learning
applications(Muntean 2002). It also appears that through gamification systems users can be persuaded to
become more eco-friendly as shown in the system called EcoIsland(Liu et al. 2011). In the next picture
(figure 3), the sensors (such as a laptop) report pollution levels to the application and the trading of e-
mission rights occurs between neighbours (users).
17
Figure 3: EcoLandPicture(Shiraishi et al. 2009)
2.2.2 Essential Game Elements
Playing enjoyment is central to computer games, although there is no accepted model of player enjoyment
in games. A recent model, however, called GameFlow, attempts to fill this void and consist of eight
elements- concentration, challenge, skills, control, clear goals, feedback, immersion and social
interaction. Concentration is about providing stimuli from different sources and assigning the user enough
workload. Games most also be sufficiently challenging and match player‟s skill level. Immersion is about
players having a deep but effortless involvement in the game(Sweetser& Wyeth 2005).
This model has been used by many including Queensland University of Technology gamification of a
mobile orientation application(Fitz-walter, Tjondronegoro & Wyeth 2011a).
2.2.3 Benefits
In 2010, the user population of Farmville was 60 million users/1 percent of the world population(Burke &
Hiltbrand 2011). This game is about users maintaining a virtual environment.
Gamification popularity did not occur accidentally as it has standard principles. For instance, community
collaboration principle states giving challenges that must be solved by a community. In addition, other
principles called dynamics are also used for successful gamification. There is the use of theepic meaning
dynamic which deals with exciting users by making it feel they are working for something
significant/awe-inspiring. Then there is the free lunch dynamic which involves giving a user a gift which
is especially enjoyable when given by a friend. Lastly, users will repeatedly use an application they find
interesting thus gamification attempts to make applications interesting which should be any system‟s goal
as well. (Burke & Hiltbrand 2011)
18
Gamificationimproves student pass rates as seen when applied to a 1st year programming class in the
University of Ulster(Ulster 2012).This will be expounded on later in the paper.
2.2.4 Limitations
By continually rewarding users in a game, you may remove the moral value of the actions done by the
user. To counter this, one needs to ask whether the user would still be interested even if they will not be
rewarded.(Burke & Hiltbrand 2011)
There are potential issues that occur when adding game elements thus proper consideration must be taken
in the process. (Fitz-walter, Tjondronegoro & Wyeth 2011a)
Also, financial cost is a key issue to consider when adding game elements to an application and can be
reduced using a generic tool for game definition and management(D. Bustard, M. Black, et al. 2011). For
this Honours project, the cost relates more to work hours thus it is important to implement only the
necessary game elements.
Lastly, as a result of limited memory on mobile phones not all gamification features can be
implemented(completely) on them(Jong et al. 2008).
2.2.5 Psychology
There are many psychological theories that make games engaging. Flow(in games), which compares from
positive psychology, is the feeling of complete and energized focus in an activity, with a high level of
enjoyment and fulfillment(Chen 2007). There a psychological reasons why gamification works. For
instance: the use of badges in gamification has a social psychology (as well as human computer
interaction) derivation. The most motivating goals are said to be those just outside of comfortable reach.
Research has shown that people will even consume physical goods in order to achieve the goals of a
game. The actual fun and adventure of goal seeking is a major reward for any user (as there is no
monetary reward). Moreover, the fact that a goal can be embodied in a game badge, the user can have
something to show to friends as a mark of achievement(in the game) (Antin & Churchill 2011).
2.2.6 Conclusion
It is clear that gamification is a helpful tool in aiding learning however scope needs to be carefully
defined in the implementation as well asconsidering other limitations.
2.3 Gamification methods at university
2.3.1 Introduction
At the highest level, the methodologies of using game design elements in non-gaming contexts is still
growing and is of keen interest in the HCI(Human Computer Interaction) field.(Deterding, Sicart, Nacke,
O‟Hara, et al. 2011c)
19
This section outlines a number of gamification experiments in education, which naturally exhibits game-
like attributes like giving points/”marks” for assignments and passing a student to another year/”level”
when ready. (Fitz-walter, Tjondronegoro & Wyeth 2011a)
2.3.2 Types
Achievement Systems
The use of a personal orientation passport for smartphones was tested by 26 university students at the
Queensland University of Technology. The system uses game achievement mechanisms like reward for
application use (adding friends), answering service related questions and so forth. Students used the
application during orientation and provided feedback each evening about the highlights/events of
orientation. It was found that 96% of users felt that the achievement concept added value and comments
given by students include “such a fantastic twist”, “was genuinely fun” and “great for killing time
productively” (Fitz-walter, Tjondronegoro & Wyeth 2011a). Here is a picture (figure 4) of the mobile
orientation application which just shows a list of the student‟s achievements:
Figure 4: Orientation app Screenshot (Fitz-walter, Tjondronegoro & Wyeth 2011a)
Leaderboard
Games can be used to enhance e-learning such as the one made at University of Ulster(Ulster 2012)in the
UK(United Kingdom). These are some of the key engagement factors: fun, social, challenge, structure
(clear objectives and constraints) and feedback. The game ran in 2007 on a first year programming class
and assigns points for the following: attendance (for lectures and tutorials), contribution to tutorial, online
revision quizzes (special highest score reward) and more. The pass rate in 2006-2007 was about 65% and
20
improved by 10% the next year. Also, in the next year about 50% of the class achieved 60% for the
course and higher. Here (figure 5) is a graph supporting this:
Figure 5: Student Improved Pass Rates (D. Bustard, M. Black, et al. 2011)
Leaderboards which list all the students ranked by points, were used for students to track their
performance and compete with each other meanwhilst also knowing that staff could also see the
leaderboad (possibly increasing students competitiveness)
Progress Bars
Firstly, progress bars are a technique of giving visual feedback to a user of their closeness to a reward.
There has also been a study looking to raise student motivation about coursework at University of Cape
Town(UCT) 2nd
Year Games Design students were evaluated and the average type of gamer was said to
be a person who likes to solve puzzle and beat competition. The other evaluation found progress bars to
be the most motivating gamifcation technique, then leaderboards but forums the least. Gamification of
forums in this context refers simply to adding a user reward system in the forums and that idea was not
rated highly by participants.Progress bars enables users to monitor their points and proximity to their next
big achievement(Donovan 2012). Here are the graphical (figure 6) results:
21
Figure 6: Comparison of Gamification Techniques(Donovan 2012)
Badges
Badges are “virtual goods” or digital artifacts that have some visual representation and are awarded to
users who have finished a particular set of activities. Badges have many uses but just goal-setting,
reputation and self-affirmation will be discussed here. Badges challenge users to meet a goal set for them.
Goal setting has been shown to be an effective motivator. Also, the reputation of users can also be quickly
determined by looking at what badge class they are (for example bonze or gold). Furthermore, since
badges mark significant milestones it can be used to inspire the user who attained it and even peers(Antin
& Churchill 2011) .Success stories of badges include Q&A site StackOverflow‟s(StackOverflow 2012)
system of badges to encourage productive participation and more in the university context, achieving a
degree can be seen as the ultimate badge/success(Donovan 2012).
2.3.3 Chosen gamification tool
As mentioned forums, are the least favorite gamification technique(Donovan 2012) but unfortunately
through a meeting with the Vula coordinator, it is quite complex to gamify on the forums. Instead, a much
similar tool (to be introduced in next section), the wiki is considered to be far more open to gamification.
The wiki is able to log all reads and writes, and it‟s also simple to extract content of the wiki to another
tool.
22
2.3.4 Conclusion
There are numerous techniques of gamification techniques, progress bars the most successful and forums
the least. This suggests may suggest that forums needs substantial gamification but this is quite complex,
so a very similar (but easier to use) tool, the wiki, is the tool of choice.
2.4 Wiki
2.4.1 Introduction
There is no definition of the Wiki Wiki Web or Wiki/wiki (in short) that is shared by all. It is a discussion
medium, a repository of ideas and a collaboration tool. In a wiki, anyone can edit or add a page using
basic syntax to write content. This means anyone can participate and leads to the site evolving in a
collaborative way(Tazzoli 2004) . The concept of a wiki(Leuf& Cunningham 2001)and the first working
application is made by Ward Cunningham in the published Portland Pattern Repository‟s Wiki in May
1995(Cunningham 2012). “Wiki wiki” is a Hawaiian term for quick. The name underlines the fact that the
wiki system should be easy to use and learn.
2.4.2 Structure
The structure of a wiki is open and evolves freely (as users add to previous users‟ work). Although
standard web pages organize the content in a predefined hierarchical structure, the wiki uses an evolving
graph. Wiki pages are nodes in a graph that users can establish links between(pages)(Tazzoli 2004) .
The problem with a typical wiki structure is that pages are created quickly and linked to other pages but
the community does not often arrange content in a meaningful way (or in category or tree like fashion).
Therefore there has been work done by project SHAWN/Structure Helps a Wiki Navigate(Aumüller
2005) which allows a user to enter relationship data in a wiki article instantaneously and easily create
meaningful relationships. Organized wiki‟s have also been shown successful through projects like
Wikipedia(Wikipedia 2012b), which is a collaborative, multi-lingual encyclopedia. (Aumüller 2005)
2.4.3 Conclusion
There is no consensus on a definition of a wiki, but it can be thought of as a collaboration tool in which
anyone can edit or add a page using basic syntax. The name wiki actually comes from the term “quick”,
which shows that it is meant to be easy to learn and use. The structure of a wiki, can also contribute to
how easy it is to navigate.
23
2.5 Gamificationof the wiki
2.5.1 Introduction
The wiki is already implemented in the Vula system(Sartorio 2009) but is an ideal candidate for
gamifciation as it is easy to manipulate according to the Vula coordinator. Sakai is an open source
learning management system which is also used by UCT to create the more specific Vula(Suleman 2008).
Sakai also has outstanding versatility for extension or configurations of tools, which will therefore apply
to the wiki tool as well(Sartorio 2009).
2.5.2 Gamifying the wiki
Wiki Gamification
The wiki can also contain progress bars by having a character that tells a user how far they are to
achieving bonus marks and canalso have a leaderboard showing all users points therefore it can
essentially include any gamifcation technique.
It is important that users can judge content, build reputation and find relevant content quickly. A study
conducted(by Palo Alto Reasearch) on social Question and Answer site called Quora Center, analyzed
data across 60 questions and 3917 users. Results indicated that students find primary (first-hand)
information posted by fellow students more authoritative and also base the credibilityof other users on
past contributions. Good content is also identified through social/user voting per answer but this can also
be biased thus combining this with more advanced ranking algorithms produces fairer results. Such a
ranking algorithm would not just consider an answer‟s votes but also the answerer‟s past contributions in
ranking. (Paul & Hong 2012)
It is also helpful to understand crowdsourcing which is defined as the outsourcing of activities (like the
answering of wiki questions) to a group of people in a community. Community based crowdsourcing, is
when the community/users are involved from analysis to implementation hence feeling a true ownership
of the project. There are strategies on how to inspire community members to contribute to a network (or
wiki in this case). One such strategy is using rewards of the monetary and non-monetary kind. Monetary
rewards are the strongest motivator to encourage community participation but are known to sometimes
have a short-term effect. Whereas, non-monetary rewards can be points allocated to user for extensive
contributions which leads to role promotion (enabling more application access). This can have long-term
effects although users must be given rewards they actually enjoy(Puah& Abu Bakar 2011).
Given this evidence, gamification on a wiki is quite viable but requires more detailed implementation
research (discussed in design section) for full viability
2.5.3 Conclusion
There are a number of gamification techniques that can be applied to a wiki. These include, using avatars
and even crowdsourcing to encourage crowd participation as quality contributions. However, it is clear
that more specific implementation literature (discussed in design section) needs to be obtained for full
feasibility for gamified a Vula wiki on mobile.
24
2.6 Mobile Usability
2.6.1 Introduction
Usability can be considered as the extent to which specific goals can be achieved with effectiveness,
efficiency and satisfaction by specified users carrying out specified tasks in specified environments (Nigel
Bevan 1995).The key terms here are:
Effectiveness( the accuracy and completeness of product)
Efficiency(how well a product uses time and resources);
Satisfaction (the comfort and acceptability of use of product)
(Nigel Bevan 1995)
Software designed on mobile phones is quite different to that designed on desktops , in that mobile
phones have limited memory(Jong et al. 2008) , screen size, limited page types and more.(Jones &
Marsden 2005, p.251)
Therefore careful consideration must be paid to applications to make them easily usable for people.
Also, one can imagine eating a chocolate cake baked by a master chef, only to find out that it must be
consumed by a straw(Jones & Marsden 2005, p.29).This equivalent to developing great technology but
having a poor user experience.
2.6.2 Techniques
Figure 7: Microsoft Word file menu (Jones & Marsden 2005, p.230)
There is the principle of least astonishment which states the icons must be consistent in functionality and
form across application. For example, if a floppy disk (figure 7) icon means save on one screen, it should
not mean exit on another screen (for instance, the save icons in the two file menus in figure 7 must be
consistent). Fitt‟s law, states the time taken to acquire a target (app icon in this case) is a function of the
distance to and size of the target. This essentially means the most frequently used items must be easy to
access by being a few clicks away from the user and vice versa. (Marsden 2012)
Hick‟s law states the time taken to reach a decision goes up as the number of choices increases therefore a
mobile phone must not have many icons on a screen. In psychology, 7+-2 items is the maximum short-
25
term memory can retain thus this is the maximum number of different icons a screen should
have(Marsden 2012).
Also, direct manipulation is the continual representation of objects of interest, rapid reversal of actions,
and the replacement of typed commands by a pointing action on the object of interests. Examples of direct
manipulation is turning a steering wheel left, by rotating the wheel and the car immediately turns
left(immediate feedback)(Shneiderman& Plaisant 2005, p.214).
Guidelines and Standards
Design standards are high authority (set by international committees) and specific rules (not general) for
software development.
Guidelines are low in authority (set by companies) and high in generality for software development so can
be applied to many platforms. There are also not many standards focusing on usability whereas guidelines
have much to say on usability. (Marsden 2012)
Shneiderman’s 8 golden rules
An example is BenShneiderman‟s 8 golden rules of user interface design(Shneiderman& Plaisant 2005).
This is also happens to be one of the most widely used guideline for interface evaluation(Sulaiman et al.
2009). The first rule is to strive for consistency. It is said that consistency is the most frequently violated
rule as it has many forms. Consistent sequence of actions must be used in similar situations, consistent
terminology for prompts, menus and consistent colour, fonts, etc. Exceptions can be made such as
confirmation pop up boxes when pressing delete button whereas other buttons do not use confirmations
necessarily.
Always giving users feedback for every action, preventing errors (and giving clear error messages) and
reducing short-term memory load (on user) are other notable rules. To elaborate, feedback can be if the
user flicks their fingers up on a touch screen phone, the page should immediately scroll down. This is
minor feedback for a minor and frequent action. However, if the user clicks the delete button, a pop up
confirmation box appears, grabbing the entire attention of the user. This is major feedback for a major and
infrequent action. Also, errors can be prevented by graying out menu items. Likewise, reducing short-
term memory load(on user) can be achieved by having “seven plus or minus two chunks” of information
on a page(Shneiderman& Plaisant 2005 , p.74).
A comprehensive set of guidelines is the 162 guidelines of Designing User Interface Software (Smith, S.
L., & Mosier 1986) . This is quite specific including rules such as using simple sentences, left-justify
columns of alphabetic data to allow for rapid scanning and for size coding, make larger symbols at least
1.5 times the height of the next symbol.
The International Standards Organization have good guidelines(and standards) on usability that add
credibility to any system the adheres to them although the standards have to be bought at high prices,
unfortunately(Nigel Bevan 2001).
26
Transferring from desktop to mobile phone
Microsoft also has guidelines for transferring existing applications from one platform (e.g. desktop) to a
smaller screen (e.g. mobile). This includes rules such as locating all unnecessary elements and removing
them, placing elements which are side by side (in desktop) to underneath each other (in mobile) and more.
Such guidelines, can be of great help in apply gamification techniques on a mobile device.(Jones &
Marsden 2005, p.79)
Menus
Menus are useful addition to any interface as they provide recognition rather than recall (this is the fact
that we can know what an object is by seeing it but struggle to remember it by memory). Menus allow for
a user to select a command from a screen rather than having to remember the name and type it via
command line(Jones & Marsden 2005, p.224). Menus also need to be designed carefully in order to be
successful(Shneiderman& Plaisant 2005 , p.268).
Several authors suggest using four to eight items per menu level and at most three to four levels for the
whole menu(Shneiderman& Plaisant 2005 , p.283).
2.6.3 Wiki UI Techniques
Wiki articles can have different icons sizes determined by frequency of use(e.g. reads and edits), to enable
users quick access to the most popular articles, by using tree maps(Shneiderman 1992a).Tree maps is
essentially a method for displaying hierarchical data by using nested rectangles(Turo 1992).The following
diagram below(figure 8) is an illustration of tree maps where(in the case of a wiki) the bigger rectangles
would represent more frequently accessed articles and vice versa. Also, this is in adherence to Fitt‟s Law,
as the items/rectangles that are biggest will take less time to access than smaller rectangles.
27
Figure 8: Tree Maps (Shneiderman 1992a)
2.6.4 User Evaluations
Designers can enjoy their own systems so much that they do not adequately evaluate them. Failure to
perform and to document testing could possibly lead to malpractice lawsuits from users due to
errors(Shneiderman& Plaisant 2005 , p.140). Mobile computing effectively creates a new range of
applications which users have not seen before(Jones & Marsden 2005, p.198). Therefore, it would be
helpful to understand how users interpret a new interface (through a paper prototype possibly) given their
current mental model of how interfaces work. This is known as a conceptual model extraction, and yields
qualitative results. (Jones & Marsden 2005, p.197). There is also the experimental process (part of
scientific method) of conducting a user evaluation which will involve having a independent variable
(system), dependent variable(like time), and falsifiable hypothesis( a testing question which is not trivial
to succeed), etc. Research also delves into other issues such as keeping an experiment unbiased and
choosing a controlled environment(required for conceptual model extraction and a user lab for the
experimental process)(Jones & Marsden 2005, p.210).
It is even possible to not include users in the user evaluation. An example is a heuristic evaluation(ten
usability heuristics), which is a technique invented by Jakob Nielsen(Nielsen 1994) for experts to evaluate
interfaces. Essentially, a set of experts/evaluators inspect the interface using only a small set of broad
usability principles. Once again, consistency, error prevention and feedback make an appearance with
recognition rather than recall among the new additions(compare to Scheiderman‟s interface design
list)(Nielsen 1994).
28
Success Factors
The technology acceptance model is a model that can help determine whether a technology will be a
success or failure. They are useful in evaluating prototypes and new systems whilst also help to see what
made a past product successful. Overall, the success of technology is depends on individual beliefs and
the influence of those around them.(Jones & Marsden 2005, p.63)
2.6.5 Conclusion
Good usability is certainly needed in a mobile device although quite tricky (but possible) to implement
considering the limitations of mobile devices. There are guidelines aplenty on how to design a good user
interface for mobile. Furthermore, there are techniques of how to translate a desktop application to a
mobile one, which is a big part of developing the new gamified wiki tool. Lastly, there are also standard
(from desktop) techniques of how to evaluate a user interface like the experimental process and new
techniques like the conceptual model extraction. These evaluations cannot be underestimated as a poor
user experience could lead to a software company being sued for malpractice.
2.7 Critical Comparison The following section critically evaluates gamification(table 1), gamificationmethods (table 2) and user
evaluations techniques (table 3).
Table 1:Gamificiation - Benefits versus Limitations
Benefits Limitations
A bestseller education application about
maintaining virtual farms has 60 million users/1
percent of the world population. Gamification is highly successful, even though it only became an
industry concept in 2008.
Adding multiple game/gamification elements
requires care otherwise conflicts may arise.
Gamification uses successful techniques such as
achievement systems which gives users rewards for impressive work hence encouraging their return.
By continually rewarding users in a system, the
moral aims of the system may be defeated.
It improved the pass rate of a 1st year university
class in University of Ulster.
Gamification can add exorbitant financial costs to a
system depending on implementation scale.
Table 2: Gamification Methods at University
1. Forums 2. Progress Bars 3. Leaderboard
Characteristic A question and answer
platform which can be
gamified through voting
Technique of giving
visual feedback to tell a
user their distance away
Technique to drive
competition amongst
users.
29
answers and more. from a reward.
Comparison Although ranked the
lowest in a survey(Donovan 2012),
has the most potential of
expanding(and so too the wiki).
Ranked highest in the
survey but can be incorporated in a
forum/wiki.
Ranked relatively
middle in the survey, but can also be
incorporated in a
forum/wiki.
Table 3: User Evaluations
1. Experimental Process 2. Conceptual Model
Extraction
Equipment Working/Interactive prototype Paper prototype/Interface sketch
Results Quantitative Qualitative
Location Normally laboratory based Any controlled setting
Comparison Essential component of scientific method and since the results are
numeric, hard to dispute.
Mostly relevant in mobile prototyping and very helpful to
get user feedback early.
Inspired from (Jones & Marsden 2005)
2.8 Gap in the knowledge Firstly, there are no actual case studies in the literature that speak about gamifying a wiki and this absence
is clear from section 2.5.2 on gamifying the wiki. Therefore, this is an area that this project seeks to tap
into since no one has done so already.
Secondly, there is no literature on using gamification to improve usability; therefore this is also a focus
area of this project. In fact, a previous study mentioned that they are sometimes at odds with each
other(Fitz-walter, Tjondronegoro & Wyeth 2011b).
Therefore, considering these two gaps in the knowledge, it would be ideal to contribute research on the
gamification of a wiki and using gamification to improve usability. Indeed, this makes up the research
question of the project, which is to: improve the usability of the Vula wiki by using gamification.
2.9 Conclusion Gamification is a powerful technique to make online learning tools more attractive to students. Results
have shown that millions of people enjoy games and that academic pass rates can be improved by making
parts of education tools more game-like. Pertaining to the Honours project, research points to
gamification of the Vula wiki as the most feasible option in terms of ease and use(all reads and writes are
logged) and reduced complexity in comparison to forum. However, one of the main limitations to
gamification appears to be the cost/extensive effort that can go into it using it. So, it is important to use
only the key elements required. Also, usability is a key aspect for any mobile application and the
appropriate guidelines and usability testing needs to be taken to produce a good user experience. Overall,
30
gamification is not a perfect solution to improving student engagement with educative material but can be
quite effective if applied properly.
31
3 Design Chapter
3.1 Introduction Designing a mobile interface poses many challenges like limited screen estate and phone memory (Gong
et al. 2004). Furthermore, gamification has been known to introduce clutter in a user interface(Fitz-walter,
Tjondronegoro & Wyeth 2011b) and indeed this was also mentioned in the project‟s proposal
presentation. However, this project attempts to provide a usable mobile interface through using a user-
centered design methodology.
This methodology is what created the mobile Front-end, as illustrated in the System Overview – the first
section to be discussed. Thereafter, the scope is defined followed by the application use cases.
Then user-centered design methodology is then explained and followed by explanation of each of its
sections. Thus, the identification of user requirements is discussed initially. Then, due to its importance,
the initial gamification requirements and their refinements (due to scope creep) are discussed separately.
Afterwards, the need for rapid prototyping and its application to the system is discussed. Finally,
design/software evaluations are discussed as they are quite integral to prototyping(Shneiderman& Plaisant
2005, p.141).
3.2 Development platform The jQueryMobile( the jQuery Foundation 2012c)framework which is an extension to jQuery, is the main
technology used for development. It simplifies parts of a Web application such as navigation, form
elements and even creating page transition effects without coding the JavaScript yourself. It also supports
most smartphone brands(besides Windows Phone 7)(Godwin-Jones 2011). With regards to languages
(scripting) used, there is: HTML5, CSS3 (Cascading Style Sheet latest standard), JavaScript, jQuery and
Ajax(Asynchronous JavaScript and XML). HTML5 is a new draft World Wide Web Consortium(W3C)
standard that is already used with jQueryMobile although it also helps developers create rich web
applications without having to rely on proprietary technologies(Vaughan-Nichols 2010). CSS3 is used as
it comes with jQueryMobile although it does have benefits of automatically resizing web content for
display on a small-screen phone (Godwin-Jones 2011).
JavaScript is used as it is supported by most web browsers (Mikkonen & Taivalsaari 2009). jQuery is
used as it is a JavaScript library that allows developers to create high quality applications with a minimum
amount of code and is cross-browser compatible(Evans et al. 2012). Ajax is useful for buffering data
before the user needs it and improving user inter-activity with an application(Zucker 2007).
In terms of development tools, notepad++ is used(for all scripting and programming)as it is a common
open-source code editor(Zhang 2012). Finally, a drag-and-drop tool for building simple UI(User
Interface) layouts, called Codiqa, is also utilized as it simplifies development and is part of the jQuery
mobile website(Holtzinger et al. 2012).
32
3.3 Scope In-scope
General Wiki functions such as create, read and edit article but optimized for mobile phone display.
A restyled commenting system for Wiki pages similar to Facebook such as a comment liking feature
since Facebook is already familiar to most students(Mathews 2006).
Ordering Wiki pages according to sections like Wikipedia(Wikipedia 2012b), as this is similar to
peephole displays where only a small window/peephole of info is displayed from a large display
area(Yee 2003).
Selecting sections as content between each heading.
Gamification techniques such as leaderboard, badges, avatars and achievement points.
Basic group functionality that allows students to join a group (only once) and view both group and
individual leaderboards.
A visualization of popular articles using tree maps as this is an effective visualization technique for
hierarchical information(Shneiderman 1992b).
Out of scope
Administrative settings such as choice of language, font size, background picture, etc.
Selecting sections as a specified continuous block of text.
Only one course catered for.
Extended group functionality such as having badges and achievements per group, and students
joining different groups to their original choice.
An interface for lecturers to personally award pages of their choice and perform other unique tasks.
Heuristics to automatically determine valid academic content.
3.4 Methodology The main phases of typical user-centered design (figure 9) are: strategy, analysis (requirements), design
(prototyping) and evaluation, then repeat until 4 iterations (due to project time constraint in this case) to
produce the final product. Strategy is about the purpose of the project as established in the proposal
already, so not included here. The remaining phases are discussed in order in the following sections:
33
Figure 9: User-centered design. (Telono 2012)
3.4.1 Clients/Supervisors
The supervisors spawned the initial project idea of students engaging more with academic learning
material. The co-researcher suggested the idea of gamification to achieve this and together with the
supervisors the initial requirements were made. This set including some of the following
gamification/reward-based techniques in the first mindmap (Figure 5) created by the researchers and head
supervisor:
Rewards for participation in forums and Wiki
Rewards for number of items download
Rewards for creating new Wiki‟s and forums.
Peer evaluation where peers review each other‟s work and the top 10 students are displayed in a
leaderboard.
34
Figure 10: First Requirements Mind map
After some more brainstorming sessions(figure 10), with some help from the co-supervisor (the
Vulacoordinator), the forum and Wiki became the initial gamification platforms, due to the simplicity of
extracting information from them. However, during the proposal presentation, it was said that improving
the forum is far easier than the Wiki and too simple for one partner to do. Furthermore, the Vula
coordinator later felt that to gamify both tools is too challenging thus refined the scope of the project to
gamifying just the Wiki tool instead. Thereafter, the supervisors continued their contribution to
requirements however the end-users (computer science students), also became involved from the paper
prototype conceptual model extractions and onwards.
3.4.2 Literature Based
With regards to gamification, the literature greatly influenced the game strategies such as the use of
leaderboards, badges and points(Deterding, Dixon, Khaled & Lennart 2011b).
Furthermore, gamification at Ulster University which led to the improvement of student pass rates
(Bustard et al. 2011), has strong influence on the system design. This study describes features such as:
35
Rewarding users as individuals as well as in groups.
A game points gauge/speedometer(figure 11) to show users their current points, as illustrated here:
Figure 11: Sample GEL (game-enhanced learning) performance gauge tool (Bustard et al. 2011)
A leaderboard with the added feature that one can click on a student‟s name and see/”drill-down into”
all their achievements
Indeed, all of these features are eventually implemented into the final gamified wiki.Also, the mobile
literature such as (Jones & Marsden 2005) and (Shneiderman & Plaisant 2005), greatly influenced the
mobile design requirements. This is because mobile interface guidelines can help improve
usability(Seong 2006) .There is much to say on what improves subjective usability ratings from past case
studies such as grouping related items to allow for quick scanning and using less dense
displays(Shneiderman& Plaisant 2005, p.497).
After, using the usability guidelines to produce the initial paper prototype, the students can then add their
interface requirements during the conceptual model extraction. Ideally, students should not have
completely diverging interface requirements as many usability guidelines are from empirical
studies(Shneiderman& Plaisant 2005, p.494)although they need to be refined for the specific
project(Shneiderman& Plaisant 2005, p.66). To follow is a bit more on the conceptual model extraction.
3.4.3 Conceptual Model Extraction
A conceptual model extraction is used to harness users‟ thoughts when dealing with an interface that is
unprecedented to them, given their pre-existing mental models of how interfaces work. Essentially, this
entails having a user explain what they think different icons of interface icons.
36
3.5 Gamification Requirements The literature greatly influenced the game strategies such as the use of leaderboards, badges and
points(Deterding, Dixon, Khaled & Lennart 2011b) . This is because these are tried and tested approaches
that have led to improved pass rates in some instances(D. Bustard, M. Black, et al. 2011). Like any
guideline, the gamification requirements became contextualized and refined by the researchers and
supervisors initially. The gamification requirements are refined after each iteration after
student/supervisor feedback as this is typical of the iterative nature of user-centered design (see Figure 4)
An example refinement in the feasibility test software, is the initial use of a simple rule that awarded
students points for logging into the Wiki. However, this is later removed since an expert evaluator sees
the rule as too simple and indeed, gamification literature does mention presenting game rules that match
the user‟s skill set(Sweetser& Wyeth 2005). To follow, the actual prototyping process for the project is
described.
3.6 Prototypes After the requirements are established, the user-centered design methodology (see Figure 4) continues to
specification and design (of a prototype). A prototype allows for designer to communicate ideas to
various team members(and users), have them critiqued and commented on, thereafter alter the design for
the next round of critique. (Jones & Marsden 2005, p.170). Prototyping offers a quick way to incorporate
user feedback into a design(Holzinger 2004) and allows one, as Scott Jenson says, to “fail fast”, hence if a
design fails enough times it will soon become right(Jenson 2002).
There are various prototypes that serve different purposes and are used at different times(Jones &
Marsden 2005, p.170). The ones chosen in this project are discussed systematically below.
The first prototype, which will be referred to as the feasibility software, is vertical software prototype that
is actually discussed in the implementation chapter as it relates more to that. A vertical prototype offers a
selection of functionality implemented in its intended form whereas a horizontal prototype shows all
intended functionality but not implemented fully(Floyd 1984). This prototype is a vertical prototype in
order to determine the feasibility of the system(Jones & Marsden 2005, p.178)as it tests whether the Wiki
can be gamified for just one rule whilst having a working back-end and front-end. Also, since this is the
first prototype, it does not resemble how the final system looks hence it is low-fidelity whereas it would
be high-fidelity if it did (Jones & Marsden 2005, p.170). The second prototype is also low-fidelity.
The second prototype is the first horizontal paper prototype but still low-fidelity, as it occurs early in the
design phase(Jones & Marsden 2005, p.171).Given that the system is now proven feasible, the horizontal
prototype is used to show all the intended functionality(as paper screens) of the system without any
implementation so that the students and supervisors understand what they want out of the system(Floyd
1984)
Indeed, a paper prototype is known to be the fastest rapid prototyping technique as it requires simple
equipment(pen and pencil) whereas program is far more time consuming to create(Rudd et al. 1996).
Furthermore, low-fidelity prototypes support fast fail, a concept introduced by Scott Jenson(Jenson
37
2002)by allowing changes to be made at little time costs. Below (Figure 6) is a paper prototype screen of
the new Wiki, showing the badges collection of a student:
Figure 12: Paper Sketch of Badges Screen
The 3rd prototype is still a paper prototype (figure 12) but is now high-fidelity. Although, it is on paper, it
is high-fidelity as it represents how the screens look in the final version/iteration (to a large extent).
The 4thprototype is a horizontal software prototype that covers nearly all the functionality but no
implementation. This is because, the later in design phase, the more the prototype should look like final
version(Jones & Marsden 2005, p.171). Indeed the 4th prototype was followed by the final system where
functionality was fully implemented.
Evaluations are an important aspect of prototyping, and these are discussed in the next section.
3.7 Evaluations At the end of each design cycle, an evaluation is done (see Figure 4). Even the most experienced
designers with all their wisdom have the humility to know the necessity of extensive
testing(Shneiderman& Plaisant 2005, p.140).
38
Also, no matter how good ones designs are, users will always do something unexpected with the
interface. In addition, choosing the correct evaluation technique depends on the type of results required,
user and equipment availability(Jones & Marsden 2005, p.195). Furthermore, the appropriate ethical
clearance and ensuring users of anonymity is also necessary to all user (i.e. student) evaluations.
The evaluation of the rough feasibility test software was a quick-and-dirty evaluation(Jones & Marsden
2005, p.197). This simply involves asking an end-user (in this case, a project marker), how well the
design (in this case, software) looks to them. Indeed, this was quite helpful as the criticism led to a
redesign of the feasibility software.
Expert evaluation by two experts (Professor Sonia Berman and Dr. Patrick Marais) was used for the
second (paper) prototype, as it can also be a natural starting point for evaluating new
interfaces(Shneiderman& Plaisant 2005, p.141). The specific type is known as a cognitive walkthrough,
in which the expert user carries out some typical interface tasks such as creating and editing a Wiki
article. This type of evaluation was chosen by the supervisor as it was the project feasibility test.
Furthermore, any system that can be learned by exploring (or does not need substantial training) such as
this one, can be evaluated using a cognitive walkthrough (Shneiderman & Plaisant 2005, p.142). One of
the chief benefits of expert evaluations is that they are less time consuming and expensive compare to
user-centered evaluations(Scholtz 2004)yet are still effective(Shneiderman& Plaisant 2005, p.141). Also
expert evaluations can be a natural starting point for evaluating new interfaces(Shneiderman& Plaisant
2005, p.141).Ideally, there should be at least 5 expert evaluators(Jones & Marsden 2005, p.208) but it also
depends on the magnitude of the project(Shneiderman& Plaisant 2005, p.142)hence 2 are used here. A
heuristic evaluation was carried out on the first paper prototype by a usability expert using general
heuristics from past experience. This led to vital feedback such as: introducing the gamification theme
from the Home screen of the Wiki and carry it through the application. The second heuristic evaluation
also led to vital feedback such as the creation of screens that never existed in the initial paper prototype.
In the third iteration a horizontal, high-fidelity, paper prototype is evaluated again using heuristic
evaluation by the head supervisor, to provide a refined interface before letting the students explore
it(Shneiderman& Plaisant 2005, p.143) This exploration is done using the contextual model extraction. It
is meant to discover what students think of the mobile Wiki interface(Jones & Marsden 2005, p.198). In
terms of ethics, students were assured anonymity and remunerated for participation. As far as the process
went, students were first given a brief overview of the system, then told to explore the paper prototype
(various screens) whilst explaining their understanding of icons. This produced some noteworthy
contributions which altered the system design for the better. These included: getting rid of the
achievements screen, as almost every student did not know what the difference between the achievements
and badges are and simplify the article editing interface to use icons(like for emboldening/italicizing),
supporting direct manipulation, rather than memorizing markdown language. Another helpful
contribution was the provision of a tutorial screen (can be deactivated at student‟s will) which explains
the Wiki gamification when the game profile icon is pressed. Ultimately, the conceptual model extraction
not only provides much needed design refinements but also helps evaluate the design based on the end-
user/student‟s expectations.
39
The 4th iteration, a horizontal software prototype, is also again tested firstly with a heuristic evaluation,
since it is a natural first evaluation of a new interface(Shneiderman& Plaisant 2005, p.141) . It is then
followed by a formative evaluation(Gediga et al. 1999) which also involves the comparison of the current
mobile Vula wiki and its gamified counterparts. A formative evaluation is used to see whether the design
objectives are being met by the students (end-users) and is also suited for development phase in order to
improve a system iteratively (Gediga et al. 1999)
The final system is evaluated using a summative evaluation, as this is a typical usability assessment of a
final design regarding guidelines, standards and other objectives(Nielsen 1993).Again, this will mainly
consists of a usability comparisons against the Vula wiki. Also, the evaluation (very formal) involves 5
users per cell/class of end-user(Scholtz 2004). Therefore, this means 5students per undergraduate
computer science year. However, it also mentioned elsewhere that 10 users are recommended per system
scenario/condition(Jones & Marsden 2005, p.212). The process consists of end-users (i.e. students)
carrying out tasks that represent core functionality (mostly) and uses metrics such as efficiency,
effectiveness and user satisfaction. In addition, a post-experiment interview is also conducted to extract
qualitative data such user satisfaction. Interviews are known to have the best value when used in
conjunction with a laboratory based evaluation (e.g. summative) and are hence used here as well (Jones &
Marsden 2005, p.205) . Also, log file analysis, which is the use of software to automatically trace user
activities (like reads and writes on the Wiki), may also be applied here. The reason is that data collection
here is cheap(gathered automatically) and it can be gathered remotely(students do not have to be in
labs)(Guzdial et al. 1994).
3.8 Conclusion Although there are challenges in producing a mobile interface that makes all the users happy, the user-
centered design methodology is able to generate valuable user insight regularly and improve the design
regularly(4 times in this project). However, gamification does not just complicate the interface design
(like the need for a tutorial screen) but also leads to the creation of new tables at the back-end. In addition,
the system architecture, tools and design methodology have been introduced in this chapter, whilst the
later will be expounded on in the chapters to come as each prototype is created and evaluated.
40
4 Low Fidelity Prototype Iterations
4.1 Introduction The use of low-fidelity prototypes enables designers to quickly see how the interface will look and get
feedback(Jones & Marsden 2005, p.171). Two such prototypes were created, referred to as software
feasibility and paper prototype 1 respectively. The initial phases of the interface design commences with
gathering the correct requirements. Thereafter, the first paper prototype is produced and evaluated.
4.2 Feasibility Study Essentially, the purpose of this study is to determine the feasibility of the system(Jones & Marsden 2005,
p.178). The software simply implemented 3 game rules that awarded users points for creating and editing
articles as well as logging in. It was evaluated by a cognitive walkthrough in which the project researchers
demonstrated the systems tasks (like creating an article) in the presence of the two project markers
(Professor Sonia Berman and Dr. Patrick Marais). After the evaluation, the following feedback was given:
When an article is created, the length of it must be checked to ensure that it is not a fake article.
There should be no points awarded for logging in as that is too simple.
Finally, the user interface should be improved so that mobile users will enjoy it.
This feedback was to be incorporated into future iterations. Also, this prototype is discussed in more
detail insection 7.2.
4.3 Paper Prototype1
4.3.1 Requirements Gathering
This system requirements were derived from the project researchers and supervisors (or clients), in order
to give users(or students) what they don‟t know they want(R. Arnold 2010). Especially since gamification
is new as the term originated only 2008(Deterding, Khaled, Nacke & Dixon 2011e),it is felt that it is best
to guide users/students with their requirements and include them in later iterations.
General Functionality
Core editing- Create, read, edit and view all wiki articles. Also, there is a Facebook-inspired commenting
functionality as many people know Facebook (Mathews 2006) .
Usability Requirements
References used include: Schneiderman‟s 8 golden rules of interface design(Shneiderman& Plaisant
2005, p.74)and Jacob Neilsen's Heuristic principles(Nielsen 2005).
For example, the consistency principle is applied from the 8 golden rules(Shneiderman& Plaisant 2005,
p.74)as the same menu bar is used throughout system. Secondly, the HCI(Human Computer Interaction)
41
laws like Hick‟s law(Seow 2005) is applied in that screens are simple enough to reduce decision making
time for users.
Game Functionality
This uses a point reward system/achievement system in which users are rewarded for liking, creating,
reading and editing articles. Users can see their achievements and badges as well as a leaderboard, to
track progress.
4.3.2 Design
Home Page
The home screen(figure 13), introduces the gamification theme from
the beginning (Sweetser & Wyeth 2005) by displaying a points
guage/speedometer. This point gauge concept is inspired from a
gamification done at Ulster university(D. W. Bustard, M. M. Black, et al. 2011) . Fitt‟s law (Seow 2005) is used in the menu in which the
closest items such as home, is the most frequently used so is closest
to the user and the game profile, is furthest away as that is not an essential wiki feature. Another noticeable feature is the treemaps of
pages, as this allows the student to quickly select a popular/lively
wiki page to read(Shneiderman 1992b).
Figure 13: Home Screen–Paper
Prototype 1
Figure 14: Create Page Screen – Paper
Prototype 1
Create Page
An essential screen this is, as it allows students to add new pages to
the wiki and helps the wiki grow (figure 14). For now, it simply consists of a title and data (or introductory content) input fields.
This screen abides by Hick‟s law(Seow 2005) as it allows the
student to decide quickly what to do as there are a few page
elements. Again, the points bar is in this screen and most, to abide to Scheiderman‟s consistency principle in the 8 golden rules of
interface design(Shneiderman& Plaisant 2005, p.74).
42
Edit Page
This page is deliberately empty as it looks to be the most challenging
of the lot(figure 15). The current Vula wiki has more than 9 icons per page, the maximum number of elements short-term memory can
hold(Ingber 2012) . However, after a heuristic evaluation discussed
below, it is recommended that the students use mark-down language for editing pages. A mark-down language is a highly human-readable
lightweight markup language.(Maeda et al. 2008)
Figure 15: Edit Page Screen – Paper
Prototype 1
Badges Page
The badges are displayed on the screen in a list format in which there
is a badge image and a short-description in each row(figure 16). This
format is inspired from the achievement screen of a previous
gamification effort of a university orientation application(Fitz-Walter & Tjondronegoro 2011). However, it is later discovered that a 2-d
grid to display badges allows for better affordance with students as
Foursquare(Foursquare 2012) use this display and it is a popular gamifed location-based service.(Lindqvist et al. 2011).
Figure 16: Badges Screen - Paper
Prototype 1
4.3.3 Evaluation
This prototype is evaluated using two heuristic evaluations, as this is a natural first evaluation of a new
interface. The first heuristic evaluation is conducted by a Masters student, in which the evaluator provides
advice on each screen based on personal opinion and Jacob Neilsen Heuristic guidelines(Nielsen 2005).
Thereafter, a second heuristic evaluation is conducted by the head supervisor, in which the prototype is
evaluated based on expert knowledge. A combination of the feedback from the both evaluations is
incorporated into the next iteration.
43
4.3.4 Findings
Home Page
Make it more gamified, such as showing a leaderboard so the student has a definite idea of the
game existence.
Search bar must be displayed top right, if the researches designer chooses to have one anyways.
Edit Page
It is suggested that mark-down language be used to allow students to bold, italicize and markup
text in general. This is basically a simplified mark-up language that allows students to perform
basic markups like embolden text, by surrounding the parts of word to be bold with underscore.
Furthermore, this mark-down will be the same as the current Vula wiki mark-up syntax for bold,
italicizing, etc.
Created/Read Page
Add a little badge for each section according to the section quality.
Add Group Page
Like the University of Ulster gamification example(D. Bustard, M. Black, et al. 2011), it will be
helpful to add a group aspect to the gamification. This allows group-cooperation and furthermore,
social interaction is a component to successful game creation.
General
A student must be able to not just like a page but also the sections of a page
There has been little focus so far (on the prototype) as to how the wiki pages will be displayed.
Emphasis must also be placed on the usability of the wiki improving even though gamification is
added.
In order to display gamification notifications, instead of using constant pop-ups (which may
irritate users) rather use a inbox icon that highlights each time a message (or gamification reward)
is received. Furthermore, Schneiderman‟s 8 golden rules mentions that users must be initiators of
actions (opening a mailbox if they want) and not responders to actions(in this case pop-ups
stopping their flow)(Shneiderman& Plaisant 2005, p.74) .
4.4 Conclusions This chapter discussed the low fidelity prototyping, beginning, with the requirements gathering from
which the functionality of the system was derived. A subset was tested on the first software/vertical
prototype. Gladly, the success of this prototype showed the project to be technological feasible. This led
to a paper prototype which was evaluated by heuristic evaluation and changes were integrated in later
iterations.
44
5 High Fidelity Prototype Iterations
5.1 Introduction The High-Fidelity prototypes described here logically follow from paper prototype 1 and eventually
mirror the final system.
Starting with the second paper prototype, its design is first discussed through incorporating changes from
its previous version as suggested by the expert evaluators. However, university students are also involved
in the evaluations now as well as the usual heuristic evaluators. Thereafter, the development of a software
prototype which incorporates these alterations is discussed and undergoes formative evaluation,
conceptual model extraction (both on students) as well as a heuristic evaluation (by head supervisor).
Lastly, an improved horizontal prototype is developed based on past feedback, evaluated the same way as
its predecessor and also marks the end of the prototyping cycles.
5.2 Paper Prototype 2
5.2.1 Requirements Changes
All the findings described in the previous section are implemented in this prototype.
5.2.2 Design
Once again, the following is a sample of the designs for paper prototype 2; the full collection is in
Appendix 11.2. In particular, the following screens discussed are all improvements from the past cycle.
Figure 17: Home Screen - Paper
Prototype 2
Home Page
The home page now includes a leaderboard of the top 10 students
(figure 17) to clearly introduce the game concept from the beginning, as suggested in the previous evaluation feedback. Also, there is also an
option to join a group, as suggested in past iteration. Finally, the search
bar is now a small search icon on the top right of screen but maybe
removed depending on user feedback. The treemaps of wiki pages is now viewable only after clicking the new icon ”wiki pages”, thus 1 step
longer to reach than before and this is Fitt‟s law(Seow 2005) in action,
as treemaps is not as important as the game elements thus harder to reach.
45
Create Page
The create page screen (figure 18) now has an extra input field to
label the introductory section to give student more control. Secondly,
there is also the inclusion of mark-down edit tips that can be found under the “edit tips” icon. When clicking this icon, the student will
see possibly familiar traditional Vula mark-up style syntax for
underling, italicizing text, etc.
Figure 18: Create Page Screen - Paper
Prototype 2
Edit Page
Likewise, the edit page(figure 19) also includes an extra field to edit
the current section name, as all content should be editable in a wiki (Leuf & Cunningham 2001). Secondly, the “edit tips” icon included
here exhibits the same functionality as create page and wherever else
it is included in the system.
Figure 19: Edit Page Screen- Paper
Prototype 2
46
Badges Page
The badges are displayed in a 2d-grid format(figure 20) just like Foursquare(Foursquare 2012) as previously mentioned, this is a
popular gamifed location based service(Lindqvist et al. 2011).
Secondly, in the software version there will also be a drill-down
option on each badge as this supports the detail-on-demand concept for users, whereby they can get more information on a collection if
wanted(Shneiderman& Plaisant 2005, p.594).
Figure 20: Badges Page Screen – Paper
Prototype 2
5.2.3 Evaluation
As usual, the head supervisor provides instant feedback on the prototype before it is exposed to the end-
users (students) for the first time. Five students were chosen, from all the computer science undergraduate
years of both genders and various ethnicity groups (a representative sample). Students were assured
anonymity at beginning of experiment and briefed about experiment. Thereafter, a conceptual model
extraction is carried out on the users/students as this is the first time they have seen a gamified mobile
wiki(Jones & Marsden 2005, p.197) . Each student is asked to explore the paper prototype by
transitioning between the screens, voicing out thoughts and asking the researcher for advice where
confused. Then, the student is asked what they think of each element on each screen, and all feedback is
noted. Lastly, the student is compensated R30 for their time.
5.2.4 Findings
The findings of these evaluations are fundamental to the system development as it actually provided the
users/students with a view of the system and its intuitiveness. The results helped provide the researcher
with rich ideas from users that led to a more usable and obvious design. However, there were also rare
cases of conflicting and at times, infeasible user ideas (given the project time frame).
To follow will be a discussion of the changes each screen in turn.
CORE SCREENS
Home Screen
The home screen must be less clumsy. This can be achieved by simply spacing out icons out
more.
It is not obvious what the application is about from looking at the home page, hence a
preliminary/introductory tutorial screen is recommended which explains the wiki in the context of
gamification.
47
Have a drill down on the avatar to see how far a person is from next badge.
The level of the character should be inserted into the actual gauge.
Clicking on the points gauge (speedometer), should send the user into the game profile screen.
If there is to be a help icon, it must be represented by a question mark.
Create Page Screen
A tutorial to let the user know that editing sections is possible(figure 21), should be included
before create page loads.
Figure 21: Create Page Tutorial Screen
It will also be helpful to have a preview button when creating a page, to see if user is happy with
changes before saving.
Edit Page Screen
Should rather use a scroll bar of icons to select markup such: as bold, italics, image (and others);
as opposed to memorizing mark-down language. Furthermore, this suggestion supports
recognition rather than recall (Nielsen 1993) hence minimize user memory load.
Choose Group Screen
If groups are predefined, a drop down combo box can be used to select group.
All Pages-View Screen
Each page needs to have a rating near it, as this facilitates for quality filtering and possibly
making it faster for the students to find the information they want(Li & Wu 2010).
48
Commenting feature
It is not clear why there are “view” and “talk” icons, on each menu header. In particular, it is not
clear, why there would be a view icon since the page is already being viewed. Therefore, just
have a talk icon visible when in view mode and when in talk mode have a view icon visible.
Game Screens
Exclusion of achievements screen
There was not a single user that could differentiate the purposes of the badges and achievements
screen therefore the achievement screen will be removed since most users naturally understood
badges.
Leaderboard Screen
Change the leaderboard title from “CS TOP 10” to “GAME TOP 10”, as this makes it clear that
the table refers to a game and not academic marks.
There is also a remark of why is there a leaderboard in the first place, but this will be clarified by
having a preliminary tutorial screen in the future.
Game Profile Screen
A tutorial screen should appear before the user first visits the game menu in order to explain the
game to the user. One user remarked, “If this is a game, where do I play”. Indeed, this tutorial
screen will appear even before enters the Home Screen, so the game theme is established right
from the beginning of the system(Sweetser& Wyeth 2005) . Furthermore, this suggestion saves
the researcher time in explaining the system to users during experiments.
See All
Browse pages by categories (like subjects) and then choose page.
Popular Pages
There seems to be an ambiguity in the term popular pages, does it mean pages that are most
visited by this user or pages that are most visited by the class. Indeed, a sub-title is to be added to
the popular pages screen clarifying that the latter interpretation is correct.
Treemaps are said to be a wonderful visual by most and must be kept although one user did find it
to be complex.
Other feedback
General Changes
An alternative means of showing notifications could be to have a page that highlights when a
reward has been unlocked.
Disable menu items when not needed in order to prevent errors. This resulted from a user‟s
statement, “I will keep clicking until something happens. I want to test the whole system”. In
particular, the user asked what would happen if the “See All pages” button is clicked whilst in
create page, and the answer is that all the data would be lost.
Restyle point gauge( the jQuery Foundation 2012a), such that the points label is inside the gauge
and also indicate the badge level near the points gauge.
49
One user informs that the message box (for game notifications) is synonymous with Vula email
notifications and prefers a page that highlights. Although, all other users also observed this same
relationship, they found it attention grabbing and useful.
It is also desirable to customize settings in the system such as font size, language and background.
5.3 Horizontal Software Prototype
5.3.1 Design
This prototype improves on the design of paper prototype 2 by incorporating the student changes. This
includes in the addition of a tutorial screen, the removal of mark-down language in edit page for user
friendly icons and more. However, the design is basically the same as the previous paper prototype hence
will not be repeated here. Although, to follow is a use case diagram that details all the functionality
(excepting commenting feature) in this prototype.
5.4 Use Cases The use cases of the gamified wiki are shown in the diagram (figure 22) below .The core use cases of the
system are general wiki functions such as create, read, edit and now even liking articles. In addition, the
other functionality supported is the gamification tasks such as checking game leaderboards, badges and
also game profile. Also, it is worth noting that the “<<extends>>” keyword in the diagram means that one
activity occurring may lead to (but not necessarily) to another activity occurring. For instance: reading of
an article may lead to commenting on it. The prototype implementation will be discussed next.
50
Figure 22: Gamified wiki Use Case Diagram
5.4.1 Implementation
Changes Implemented:
All changes are implemented (except a few) but here is a sample of the changes implemented:
Having only the badges screen and eliminated the achievements screen.
Appropriate menu disabling as this prevents students from making mistakes, a form of error-
prevention(Shneiderman& Plaisant 2005, p.74) .
Tutorial screens have also been included as this facilitates quick learning of the system (Shneiderman
& Plaisant 2005, p.74) and suggested by user.
A refined edit page screen is implemented without using mark-down language but rather a scroll bar
of icons for the markup, as suggested by users and this supports recognition rather than recall(Nielsen
2005).
51
Changes Not Implemented:
No help icon included but tutorial screens are used instead. In game literature, user help can be in the
simple form of a tutorial(Sweetser& Wyeth 2005).
Commenting system not done at all, due to time constraints.
Message box icon as a page that highlights not implemented as most students found the original
message box icon useful in alerting them of notifications(for gamification).
The drill-down on avatar is not implemented as this is only an additional request by one user and not
within current time constraints.
5.4.2 Screenshots
The following screenshots (figure 23) are images of the horizontal software prototype. Initially, the user
will first see the game help page, which will explain the basic gamification aspects of the wiki (like
getting points for creating pages and so forth). Then, they will get to the home page(after pressing
continue) and in order to create a page for the first time, they will read the create help screen. Thereafter,
they will create a page and can view it. Furthermore, it is worth noting that the menu buttons are
sometimes disabled in certain screens as this is a type of error prevention that was motivated by a
previous user‟s comment.
52
Figure 23: Horizontal Prototype Screens
Having presented the prototype, the evaluations and findings will now be discussed.
5.5 Evaluation and Findings
5.5.1 Test Subjects
Once again, a representative sample of five students was chosen across all the different computer science
undergraduate years of both genders and various ethnicity groups. Just to re-iterate, the gamified wiki is
only targeted at undergraduate computer science students. Also, the subjects were picked by direct
approach and beyond being guaranteed report anonymity they were also compensated R30 per
experiment.
5.5.2 Permission and consent
An application had to be made to student affairs at the University of Cape Town (UCT) in order to
conduct research on UCT students. In addition, an ethical clearance application was approved from the
Ethics Committee of the Faculty of Science given that anonymity is assured for all participants. Not to
mention, consent was obtained from the students themselves as they signed a consent form before each
evaluation.
5.5.3 Methodology
The evaluation methodology for this iteration is a combination of formative evaluation(Gediga et al.
1999) and conceptual model extraction. (Jones & Marsden 2005, p.197).In addition, the head supervisor
also provided expert opinion on the interface (i.e. heuristic evaluation). Overall, the entire evaluation is
53
spilt into navigation and user satisfaction/engagement issues c whilst attempting to extract quantitative
and qualitative data.
As mentioned in the design chapter, a formative evaluation is used as it is able to evaluate whether a
prototype at a given iteration (i.e. formative stage) is meeting system objectives(Gediga et al. 1999)
instead of waiting till the end of the project when mistakes cannot be rectified (Landau 2001).
Furthermore, it is used to compare usability and engagement (Di Bitonto et al. 2009) like in a comparative
study of two systems(Koenemann-Belliveau et al. 1994), namely: the current Vula mobile wiki and the
gamifiedVula mobile wiki. Indeed, the learning effect needs to be alleviated by alternating the order of
which system is evaluated first for each user. Also, a conceptual model extraction is used since this is the
users‟ first interaction with such a mobile interface(Jones & Marsden 2005, p.198) and it exposes
misconceptions the user has about the system. The formative evaluation tests usability/”ease of use”,
perceived value of system and finds unintended system consequences (Landau 2001). On the other hand,
the conceptual model extraction provides insight on the students‟ understanding of system terminology.
To follow is a description of the evaluation proceedings (table 4).
5.5.4 Structure of evaluation
The evaluation consisted of a navigation and user satisfaction/engagement test (Landau 2001). First, the
student is briefed about the experiment, assured ethical anonymity and that it is the system and not them
being evaluated. This is followed by a navigation test(using formative evaluation), in which the student is
asked to carry out a set of tasks(for both current and gamified wiki on mobile) where the number of clicks
and error rates are measured (Hartson et al. 1996) using a Microsoft document table. These tasks are to
create, edit and view a page on a wiki. Thereafter, the student is asked to explore the system screens
whilst giving an explanation of system icons and raising misunderstandings/suggestions as they go along
(Jones & Marsden 2005, p.198) . Finally, a short post-experiment interview takes place in which the
student is asked a set of questions inspired from Valeria Landau(Landau 2001), which checks how well
system objectives are achieved. These are:
Determining the value of the gamifed wiki to students
Discovering any confusing terminology/places where it is not obvious what to do
A follow up on anything interesting/”unintended usage” of the system observed from user
Comments about the visual appeal and readability of the current mobile Vula wiki and the gamified
one.
Any other comments on graphics used and if there is an information overload(i.e. is the system simple
enough)
Comments about the engagement of the current mobile Vula wiki and the gamified one.
Identify any other likes or dislikes the user has of the system.
Any final recommendations on the system
Table 4: Evaluation Plan
54
Title Description
Evaluation Objective Quantitative
Navigation- is the difference in navigation speed between the current mobile Vula wiki and the gamifiedVula wiki. This is measured by number of clicks for sample
tasks completion and number of errors experienced in doing so.
Readability- is there a difference in readability between the current mobile Vula wiki and the gamifiedVula wiki. This can be tested using Tullis display-
complexity metrics(Tullis 1984) but is not done as the appropriate software could
not be found.
Qualitative Engagement- is there a difference in the engagement between the mobile Vula
wiki and gamifiedVula wiki. This is measured by post-interview questioning.
Readability- do the students find the readability between the two mobile Vula wikis different and this can be measured by post-interview questioning.
Evaluation Structure Mentioned already
Time Per User 20 minutes(although is often longer)
Venue All tests conducted in Honours Lab and one in residence room.
User Group Computer science students from each undergraduate year and 4 race groups (of
both genders) represented.
Evaluation Method Formative evaluation (navigation test between wikis) and conceptual model extraction.
Equipment Mobile phone emulator(Butts & Cockburn 2002) for gamified wiki (as the
software was not fully mobile ready) but a mobile phone for Vula wiki, as there
no mobile phone emulator for the current Vula wiki.
Recording of data Data is recorded on a nearby computer as the experiment takes place using a word
document (for each user), which has predefined Microsoft table (recording
number of clicks, etc.) and set of interview questions.
Analysis of data Data is aggregated from each document, manually by the researcher into this report.
5.5.5 Findings
Perceived Value of the System
One student mentioned that there is value in the gamified mobile Vula wiki so long as the points rewarded
can actually result in real marks or even deadline extensions. Another student, said since they never really
considered ever using a wiki thus they did not initially see a need for it. However, the same student
mentioned that it is so much easier to use the gamifiedVula wiki to create and edit pages, but again the
gamification aspect did not add more value. Yet another student also found the new wiki useful, says that,
„at least one can interact with the system even after sharing notes with peers‟. Also, a student mentioned
that it is ideal to use a Vula wiki since they do not have to bother entering their internet details as it is
local to UCT. Furthermore, this student also said the game elements add to the enjoyment of the system
but did say the interface is not obvious especially as this person is not used to wikis and does not use Vula
frequently.
Ease of Use- Navigation
55
This begins by evaluating the difference between the current and gamified mobile Vula wikis (table 5 and
6). The users are asked to carry out set of tasks (create, edit, and view wiki page) whilst their number of
clicks (i.e. number of screen transitions) and errors is measured. The results show the following:
Create Page
Not a single user is able to create a page on the current Vula wiki as they do not know the markup
required. Whereas, for the gamifiedVula wiki, the users could create a page naturally although did get
confused with the terminology in “Place Page” screen.Also, the number of clicks taken to create a page
on the current mobile Vula wiki is undefined, since no one could do it, but should be 2(if they did).
For the gamified mobile Vula wiki, the clicks is 3(by everyone) but can be reduced to 2(in the next
prototype due to a user recommendation discussed later).
Edit Page
The edit page is 2 clicks for each mobile Vula wiki, as users were able to do this successfully for both
without errors.
View Page
The view page (for a page stored in root/home page) is 1 click, for the current mobile Vula wiki but 2
clicks for the gamifiedVula wiki due to extra gamification elements. Likewise, all users were able to view
pages successfully on both wikis.
Table 5: Navigation Results – Current Wiki
User1 User2 User3 User4 User5
Create Page-
Errors
Unsuccessful,
students did
not know markup.
Unsuccessful,
students did
not know markup.
Unsuccessful,
students did
not know markup.
Unsuccessful,
students did
not know markup.
Unsuccessful,
students did
not know markup.
Create Page-
Click
2
(But not achieved by
any student)
2
(But not achieved by
any student)
2
(But not achieved by
any student)
2
(But not achieved by
any student)
2
(But not achieved by
any student)
Edit Page-
Errors
Edit Page-
Click
2 2 2 2 2
View Page-
Errors
View Page-
Click
2 2 2 2 2
Table 6: Navigation Results- Gamified Wiki
User1 User2 User3 User4 User5
Create Page- 1-Place Page 1-Place Page 1-Place Page 1-Place Page 1-Place Page
56
Errors
Create Page-
Click
2 2 2 2 2
Edit Page-
Errors
3(add section)
1( CLICK
EDIT)
3(add section)
1( CLICK
EDIT)
3(add section)
1( CLICK
EDIT)
3(add section)
1( CLICK
EDIT)
3(add section)
1( CLICK
EDIT)
Edit Page-Click
2 2 2 2 2
View Page-
Errors
View Page-Click
2 2 2 2 2
Ease of Use- Navigation Problems
At this formative stage, there are quite a few, for the gamifed mobile Vula wiki. The one that most of the
students detected is that it is not obvious that many pages have vertical scrolling as there is no fading
effect or scroll bar present. Also, many students feel that when the menu is fully disabled, it should rather
not be shown as this saves screen space. Furthermore, sometimes too many menu items are disabled
leaving the student stuck on the same screen. Yet another problem found is that after a page is created and
the student presses the back button, and tries to recreate that page, the save button simply gets stuck.
Therefore, after a page is created, the student should not be able to go back to edit the content but rather
click on the clearly visible edit icons.
Ease of Use-Readability
The current mobile Vula wiki is said to be less readable by most users than its gamifed counterpart as one
user describes the Vula icons as, „being so small that they must but be poked‟. However, the icons on the
gamified interface have also been described as too big, so need to be reduced.
Understanding System’s Terminology
Create/Edit Page
It is quite clear and simple how to create a page on the gamified wiki. Whereas with the current
mobile Vula wiki, “the edit page is not clear at all. The experience is terrible”, says one user.
Although, in all fairness, the co-supervisor (Vula co-coordinator) has mentioned that the current
Vula mobile interface is not optimized for mobile.
Tutorial screens
For the dialog box to show page again, the toggle options should not be “On or Off” but rather
“Yes or No”
Edit Page
Does the save button mean save as a draft or a published page. Rather, use the word, “publish”,
instead.
57
Group Page
This could possibly be called “team” page, as a student confused the groups to be grouping of
pages and not users.
General
One student remarks: “This mail, what is that about?” Although, most students said it is eye-
catching and they see it as an email inbox.
System’s Unintended Consequences
The system seems to be more greatly appreciated for simplifying page creation, editing and wiki
navigation than its gamification elements. Also, one person did mention that on Vula, students can
already share resources by creating Vula groups but did mention that the wiki could be useful if published
externally for access by even other universities and the public. In fact, the student even provided a link to
a South African encyclopedia called immunopedia(Immunopedia.org 2010) which promotes knowledge
and research in the field of HIV immunology in South Africa. Likewise, the wiki could also be eventually
used for national benefit like immunopedia(Immunopedia.org 2010) .
Likes/Dislikes
The points bar is generally admired by all as well as the easy navigation to create and edit pages. Also, the
icons are said to be quite big and the point bar shows up at too many places. Another student enjoys, „that
one does not need a touch pen (referring to previous mobile Vula wiki) to navigate (in the gamified wiki)
but it does need some polishing up and bug fixing.‟ Also, the student appreciated the fact that one need
not learn markup to use the gamified wiki.
5.5.6 Changes recommended
Home page
For the points bar, it would be better to have the label for the points bar inside the image(figure
24) and on the right hand-side of the picture, have the user‟s level and position (allowing quick
information retrieval). In fact, the student even volunteered to embed a beautiful label inside the
points bar image, which should be incorporated into the next iteration.
58
Figure 24: Points gauge( the jQuery Foundation 2012a) redesigned by user
Create Page
Place page is not necessary for every student, one student mentioned:” I really do not mind were
my page is placed. I just want to create a page and move on.” Therefore, after each time a page is
created, only at the end the student will be asked in a dialogue box, to place the page in a custom
region.
Created Page
Home page is very similar to created page but it would be clearer to students if they are
differentiated more.
Created page, should be simpler and not have all the gamification scaffolding such as points bar
as this distracts from the page‟s function.
Here is a screenshot illustrating this(figure 25):
59
Figure 25: New Created Page redesigned by user
The like buttons have also changed positions according to the diagram (figure 25).
Create(Page) Help
There should be either a “Cancel” button or even enabling some menu options like the home
screen to allow the user to move back.
Create/Edit Page
There dialog box that appears after the cancel button is pressed, should not appear below the
cancel button but above as it is not easily noticeable when it appears below.
Editing scroll bar- If clicking on the “bold” icon, it must actually highlight it and the icons must
have a visual affordance(Jones & Marsden 2005, p.67)such as bold icon in bold text, italics in
italicized text, etc.
Make “save” button, “publish” rather just for clarity.
Place Page
Have search bar at top right.
See All Pages
Have a filter that allows a student to see/browse just their own pages so they can make quick
edits.
Tutorial screens
For the dialog box to show page again, the toggle options should not be “On or off” but rather
“Yes or No”.
Choose group:
Possibly use a plus icon to represent adding a new group.
Profile
A beautiful effect maybe to allow student to swipe across characters horizontally by swiping
across on the touch phones.
Tutorial
Explain the menu icons at introduction screen.
General
There should be something to show that scrolling is possible. This can be just a fading/cutting off
of a button to indicate that there is more content below.
Only have the points bar in the home screen, as the user will know that they can always find it
there. This will save screen space in all the other screens and minimize user distraction. The
student, used this description: “ In music you have a catchy part of the song(called a lick) that
only occurs in a few places however if it were to occur in all the places, it would lose its
effect(referring to the points bar in this context)”
Make icons smaller, put message box in the menu header and let it highlight when rewards
notifications are necessary.
Leaderboard should only be accessible at home page, since students find it there anyways hence
the game profile menu will consists of: “Profile, Your articles, and Badges”.
60
Game profile screen should rather be called Profile, as the game information is already available
in home page.
A future possibility can be the ability to group pages into categories like computer science,
biology, etc.
Potentially have article quality control such as each page must have at least one URL link.
Head Supervisor Feedback
Change terminology of average rating to rating
The dividing lines between sections on a page are too small.
It must be obvious that scrolling is possible(just as students indicated)
Icons must be made smaller to better utilize screen space.
There should be no liking of pages but rather liking of sections as it may not be intuitive to the user
whether if they like a page, they are liking all the sections in the page or not.
The menu item called “see all pages” should be changed to “wiki pages” or “wiki” as this is clearer.
Meeting of System Objectives
Quantitative
Navigation- The gamified wiki is simpler as users cannot create a page on the current wiki
however the number of clicks taken to create a page and view a page is more than current Vula
but only the clicks for page creation can be improved as the other is a consequence of
gamification addition.
Readability- Not accessed due to lack of time
Qualitative
Engagement – The gamified wiki is said to be more exciting by most participants.
Readability- The gamified wiki is said to be more readable but needs a reduction in icon size.
5.6 Conclusion In the past chapter the high fidelity prototypes were discussed. It begins, with the integration of the
changes of the first paper prototype into the second one and evaluating (with conceptual model
extraction) the design on students (the actual end-users) for the first time. Indeed, this was very useful
feedback like that lead to the inclusion of helpful tutorial screens and a much simpler edit page interface.
Thereafter, a horizontal (software) prototype is implemented with all the recommended changes from the
paper prototype evaluation besides a few mainly due to time constraints or importance. In contrast, this
prototype uses a formative evaluation as well as conceptual model extraction producing results that
showed it to be quite usable compared to the previous Vula wiki on mobile. However, there are yet still
some improvements to be made such as eliminating the over-use of the point gauge( the jQuery
Foundation 2012a) element and reducing icon sizes. Thus the feedback from this is to be used in the final
system iteration discussed in the next chapter.
61
6 Game Rules
6.1 Introduction A component (to a minor extent) of the new Vula Wiki is that it is gamified to increase the engagement of
students with it. This chapter starts by describing the final game rules of the system, by first motivating
their inclusion. Then, the badges and avatar levels in the complete system are presented. Thereafter, the
rules not used in the system are shown. These rules were derived at the first prototyping stage but due to
time constraints were not implemented
6.2 Final Game Rules To follow is the game rules/achievements list in the system. Firstly, the usage of an achievement system
is inspired from Queensland university orientation gamification example(Fitz-Walter & Tjondronegoro
2011). Predominately, the GameFlow criteria for player enjoyment in games(Sweetser& Wyeth 2005), is
used define the game rules. In fact, this is the same basis that the Queensland university orientation
game(Fitz-walter, Tjondronegoro & Wyeth 2011b) used as well. The following is a brief list of some of
the motivations for the game rules below:
Firstly, the rules are arranged according to difficulty(from easiest to hardest) conforming to the
concept that games should have different levels of challengers for different users(Sweetser& Wyeth
2005).
Also, users must also be awarded appropriately for their efforts therefore the tasks have been
allocated points in this regard. For instance, a user is only awarded 1 point for logging in(as opposed
to removing this rule as suggested in feasibility study feedback), but 5 points for editing an article and
even awarded 20 points for creating an article.
Then, there is also the aspect of social interaction that the game supports between users(Sweetser&
Wyeth 2005) such as users being able to approve of each other‟s articles by using a “like
button”(Mathews 2006).
A leaderboard is also used as it encourages social competition amongst players(Sweetser& Wyeth
2005) and made popular by location-based games such as Foursquare(Foursquare 2012).
In terms of the format of the achievements list table (shown below) this is inspired from two sources
mainly. Firstly, the user actions (which describes a task) and achievements (points needed to unlock
tasks) fields are again inspired from Queensland university orientation application(Fitz-Walter &
Tjondronegoro 2011). Furthermore, the last field, “maximum points per student”, is inspired from
gamification at Ulster university(D. Bustard, M. Black, et al. 2011). However, for this field, instead of
just having a permanent maximum points for activities like in Ulster‟s case, it was decided to allow
this maximum to be reset daily otherwise students would no longer be awarded for their
efforts(Sweetser& Wyeth 2005).
Now that the achievements list(table 7) has been motivated(somewhat) , to follows is a list of all that
are used in the gamified wiki:
62
Table 7: Achievements List
Difficulty User Actions Achievements Maximum Points
Per Student
1 Level 1
Create a wiki page 20 points for each
page created with a
maximum of 200 points for creating
10 articles.
Maximum 200
points Per Day
2 Create a wiki
section
10 points Maximum 200
points Per Day
3 Edit a wiki section 5 points. Maximum 100
points Per Day
4 Read a wiki page 10 points. Maximum 10 per
day
5 User Login 1 Point Maximum 1 per
day
6 Level 2 Editor Receive a
like on wiki section.
15points per section
vote
Maximum 200
points Per Day
7 Author (creator)
receive like on a
section
10 points per
section
Maximum 200
points Per Day
8 Like a wiki section 10 points per
page(but voters
need 50 points minimum)
Maximum 100
points Per Day
7 Level 3 Users with top 3
points in the
leaderboard
These are the main
medals and are
discussed in badges section below.
6.3 Avatars The following is a list of the main avatar/character levels (table 8) to the system. Indeed, having avatars in
a game is known to make it more fun(Liu et al. 2011)and is also an idea encouraged by the project
supervisor. Furthermore, the avatar names are derived from the Iron Reals Entertainment(Iron Realms
Entertainment 2012).
Table 8: Avatar list
Level Avatar Name Points Needed
1 Newborn 0 (default)
2 Harmless 50
3 Courageous 100
4 Mighty 200
5 Legendary 400
*Each of these appears as characters in the “profile screen” of the wiki.
63
6.4 Badges Badges, whose names are inspired by question-and-answer site StackOverflow(StackOverflow 2012), are
also used in the system(table 9) as they help users with goal-setting as they will work towards them and a
hosts of reasons are in the literature (Antin & Churchill 2011a). Again, players are rewarded according to
effort (Sweetser & Wyeth 2005) as a gold badge is worth the most points and an editor badge the least. In
terms of organization, badges are grouped by wiki functionality (author, reader, etc.), general badges
(admirer, popular, etc.) and main achievements (bronze, silver and gold).
Table 9: Badges list
Badge Name Achievement Points Gained
Author Create 1st article 10
Reader Read 1st article 10
Editor Edit 1st article 5
Admirer Like 5 articles 10
Popular Articles received 5 likes 10
Contributor Contributed in 5 articles 10
Bronze User is 3rd
in the leaderboard (user should
have a minimum of 150 to achieve this)
20
Silver User is 2nd
in the leaderboard(user should have a minimum of 175 to achieve this)
25
Gold User is 1st in the leaderboard(user should
have a minimum of 200 to achieve this)
30
6.5 Rules not used The initial game rules (developed by the main supervisor and researchers at the first prototyping stage)
consist of mainly what is in the final game rules but with some additional rules (which are later removed
due to time limitations). Therefore, initially it was imagined that users could be rewarded points for the
following:
Adding a citation to articles
Commenting on a wiki
Answering/asking questions on a wiki
However, these rules could not be implemented as they would require unnecessary system modules to be
made (like a question and answer platform).
6.6 Conclusion This chapter discussed the game rules of the wiki. It begins with a motivation for the final game rules
used in the system, which has its foundation in the GameFlow criteria for player enjoyment in
games(Sweetser& Wyeth 2005). Then, the list of badges was described followed by the avatars. Also,
inspiration was drawn from StackOverflow(StackOverflow 2012) to come up with badge names and Iron
Reals Entertainment(Iron Realms Entertainment 2012) to come up with avatar names. Lastly, some of the
64
initial game rules which have been discarded (due to lack of time) such as points for question and answers
were described.
65
7 Final System Design and Implementation
7.1 Introduction After several prototype iterations, the system is now in its final cycle/stage. However, before describing
the final system, this chapter starts where it all began (an initial feasibility study). Therefore, the design,
evaluation and feedback of the initial feasibility software are discussed. Then, the final system is
described, which although did not follow on chronologically from the feasibility software, essentially uses
the same technology. In addition, the final system will be described (in a top-down manner) by its system
architecture, methodology, tools and platform then by screenshots. Lastly, the different changes
implemented and not implemented are mentioned.
7.2 Feasibility Study
7.2.1 Design
Interface
There is very little design consideration given to this prototype as its objective is to prove the
technological feasibility of the project. Although, certain HCI principles are adhered to such as
Schneiderman‟s consistency principle as all menus are consistent (Shneiderman & Plaisant 2005, p.74)
and there are not more than 9 items in any screen(Ingber 2012) .The following is the log in screen(figure
26) made with jQueryMobile framework( the jQuery Foundation 2012c):
Figure 26: Feasibility Software Log-In
Gamification
In terms of game rules, the student is awarded (for now) 10 points for each log-in and 20 points for each
page creation. Indeed, this is in accordance with the concept of rewarding users according to effort
(Sweetser & Wyeth 2005) , as creating a page is more effort than logging in. Also, a user is able to view
their points on a leaderboard (which has two fixed users at this stage).
66
The next section will discuss the implementation a bit further.
7.2.2 Implementation
Development Platform
This is a cross-platform development using(on the front-end) jQueryMobile( the jQuery Foundation
2012c) framework, HTML5 and JavaScript. jQueryMobile is used as it supports most
smartphones(besides Windows Phone 7)(Godwin-Jones 2011). HTML5 is chosen as it is the latest HTML
standard and it enables rich application development(Vaughan-Nichols 2010). For more, please refer to
section 3.2.2, as exactly the same technologies are applied here as well.
In addition, in terms of development tools, notepad++ is used(for all scripting and programming) since it
is a common open-source code editor(Zhang 2012)
Data Storage
Currently, all data storage which consists of user profile and leaderboard, is stored in a text file, but will
eventually be stored in a database.
Integration
There is also integration between the back-end and front-end since the application is viewable on mobile
(front-end work) yet changes (like page edits) are also reflected on the real Vula wiki, which can be
viewed on the desktop (back-end work).
7.2.3 Evaluation
Initially, before the software was fully developed, a quick-and-dirty evaluation was done by a project
marker on a basic mobile wiki. The person simply provided their opinion which led to significant
restyling of the design although not very usable yet. After much more development, the now complete
prototype underwent a cognitive walkthrough, a type of expert evaluation in which the expert simulates
users carry out typical interface task(Shneiderman & Plaisant 2005, p.142). Also, as this was the official
feasibility test , it involved the head supervisor and second-reader observing typical interface tasks such
as creating and editing pages in a wiki being performed by the project researchers(to save time).
7.2.4 Feedback
After the expert evaluation/feasibility test, the following feedback was provided:
When an article is created, the length of it must be checked to ensure that it is not a fake article.
There should be no points awarded for logging in as that is too simple.
Finally, the user interface should be improved so that mobile users will enjoy it.
67
This feedback was to be incorporated into future iterations. Indeed, this was a throw-away prototype
and was superseded by a paper prototype (which ultimately led to the horizontal software prototype).
After, the horizontal software prototype, the final system iteration described next, was developed.
7.3 Final Design
7.3.1 System Architecture
Figure 27: System Overview
Model
Database
Sakai
Controller
HTML
Representation/
Mobile User
Interface
Embedded post
(request)
function calls
Libraries
AJAX
(JSON/HTML)
PHP External JavaScript HTML Storage
Plugins AJAX-
XMLHttpReques
t
Request
View
JQueryMobile
Game Rules
JQueryMobile GameiRules
Rules
Front-End (Deon Takpuie)
-
Back-End (Reitumetse Chaka)
68
Explanation
The system architecture is illustrated above (Figure 27). It shows a separation between the front-end and
the back-end. Indeed, not both will be discussed as the researcher is responsible for the front-end for this
project.
The researcher contributed the following modules:
A game rules module, which is a set of rules in plain English implemented throughout the entire
system. Therefore, the front-end and the back-end can apply these game rules independently by the
front-end changing the screen layout accordingly and calling either dummy or real functions, that the
back-end will be preparing as game functions.
A mobile interface that is developed with jQueryMobile(more in subsequent paragraphs), which
automates the creation of HTML (Hypertext Markup Language) code with a drag-and-drop GUI
(Graphics User Interface) creator. It is also built with a jQuery framework, which is the enabler for
request and view Ajax messages between the back-end (controller) and the front-end user interface
(HTML).
Communication between back-end and front-end:
The client/front-end mobile interface receives a request from the user (maybe to create a Wiki page),
this is sent to the server (controller) as a Ajaxrequest (by JavaScript), which the server will process
with its back-end elements and return Ajax response in JSON (JavaScript Object Notation) or XML
(Extended Markup-Language). Thereafter, the client transforms this JSON or XML object into a
HTML page, again with JavaScript, so it is viewable as a mobile screen.
7.3.2 DB-Schema
69
Database Schema
The following figure below is the database schema:
Figure 28: Database Schema
The following tables (figure 28) will be populated by events invoked by the mobile interface (or front-
end) but will be created by the back-end. Firstly, on the gamification side, a badge table is created to
represent the different badges a user has and can get. Also, a leaderboard table is created for students to
view their ranking against each other through a mobile phone.Whereas, there is a group leaderboard for
students to view their rankings by group and group table is created to hold information of all groups. On
the traditional Wiki side, a page table (which contains all articles) is also created to simplify gamification
manipulation at the server-side (or back-end), and the front-end needs to be altered accordingly to include
a like button and article ratings.
70
7.4 Implementation
7.4.1 Methodology
As mentioned before, the user-centered design methodology has been used to create the system
iteratively. The process began with paper prototypes, then to vertical and horizontal prototypes, now into
a complete system. Again, the implementation of the final system began at the horizontal prototype stage,
when the windows navigation diagram (figure 34) was drawn of the entire system see appendix 11.4.
Furthermore, another version of this diagram has Portable Document Format (pdf) annotations of the
exact Ajax request and responses expected from the server and client/front-end. However, there was no
integration between the client and server for the horizontal software prototype. Thereafter, in the stage of
the final implementation, a model-view-controller framework(Krasner & Pope 1988) was used to allow
more development convenience. Essentially, the integration was not as straightforward as imagined since
the back-end was not designed as the front-end expected it. Therefore, to reduce integration time, the
back-end programmer developed both server responses and client request according to the system
diagram see appendix 11.4, so the front-end programmer merely needed to call the functions required to
perform tasks. To elaborate on the model-view-controller framework, there file with the client requests is
controller.js (JavaScript) and it communicates with the corresponding back-end controller files. All the
front-end/”view” has to do, is to call a function(such as create page) in the controller.js file, with the
necessary parameters(such as page title and name) and the controller will populate the database/”model”
with the necessary data. Furthermore, through the use of the model-view-controller framework, issues
such as maintainability and portability become more easily solvable.
Maintainability, Portability and Scale
In terms of maintainability, the system as a whole is easy to maintain, since the model-view-controller
framework allows one to add more views/”mobile interfaces” in new files whilst separately developing
controller functions(at the same time) to cater for these new mobile interfaces. Also, the front-end is also
easily maintainable since items such as: the menu, leaderboard (and more), are separate includes files, so
they can be re-used/”inherited” wherever needed. In terms of portability, the system can be re-
implemented in another web language(for example) by following the system diagram(see appendix
11.4/figure 34), and also, due to the use of jQueryMobile the system is compatible with an array of
Smartphone platforms such as Samsung,Blackberry and Apple(Godwin-Jones 2011).
Limitations
The gamified wiki was not comprehensively tested for scale, as only 10 users participated in the
evaluation but it is imagined that the model-view-controller framework provides a good foundation and
furthermore, this is not part of the front-end scope. Secondly, in terms of robustness, the software was
tested using white-box testing such as varying inputs and black-box testing(Beydeda& Gruhn 2001), such
as testing functionality. However, the software (interface) could possibly be more robust but this is not a
focus of a design/front-end project.
71
7.4.2 Tools and Platform
Again, the framework used to develop the mobile user interface is the jQueryMobile accompanied with
Ajax and JavaScript programming. Also, the development environment used was notepad++(Ho 2012)
and the mobile phone used for testing the application is: the Samsung Galaxy Ace (sponsored by the
company). For more on this section, please see section 3.2.
7.4.3 Screenshots
The following are some screenshots (figure 29) of the system, the first set of pictures cover general wiki
functionality whilst the rest cover gamification elements (figure 30). Although, there are still some game
features even in these pages like a points gauge( the jQuery Foundation 2012a) in the home screen and the
badges and like button in the created page (or the rightmost screen). This first set of pages show how a
page can be created on the wiki, when starting from the home screen. Also, the create page tutorial is
omitted (as the user can turn it off) and the log-in screen is also omitted, as it a prerequisites for all
screens. Finally, it is useful contrasting the following screens with the previous horizontal prototype‟s
version (see section 5.3.3). Indeed, one of the main changes is in the home screen, the choose group icon
is now visible and even the created page screen is 1 screenshot here instead of two. This illustrates (in a
way), that the final implementation takes far less screen space.
Core wiki
Figure 29: Final Create Page Scenario
Gamification
72
Figure 30: Final Gamification Screens
The above are gamification screens (only some). Firstly, from the game menu, one can either view badges
and drill-down further into individual badges, just like in the location-based game Foursquare(Foursquare
2012) or they can see their profile page, which shows their current avatar(as well as the info needed to
unlock other avatars) and even shows their rewards/achievements so far(this list is always the same
though).
73
7.5 System Changes After feedback from the previous horizontal software prototype, a few changes were implemented.
Furthermore, the front-end and back-end have now been fully integrated for all essentially system features
except for some optional gamification ones. Also, there has been general system refinements made by the
researchers for the benefit of the users.
7.5.1 Suggested changes not implemented
Home page
Firstly, from the previous iteration a user had suggested a beautiful restyling of the points gauge(
the jQuery Foundation 2012a), illustrated with Figure 24 , by embedding a label inside the
image, however this could not be implemented due to incompatibilities with the points gauge
software library( the jQuery Foundation 2012a)
Create/Edit page
Also, making the edit icons (such as bold or italics) highlight when selected was not
implemented as the functionality of emboldening, italicizing, underlining (and the rest of edit
icons) were not implemented as those buttons are not functional, in any case. This is because this
task is not worth the effort for the current project but would be a necessary future work if the wiki
was deployed properly, with students using it daily and needing to perform the text mark-up they
are used to.
General
There are no special effects to visually show that scrolling is possible as techniques required to do
this added unnecessary complexity to the system(Buchanan et al. 2001). Furthermore, there is
now far less scrolling needing in the system as the pictures sizes have been reduced, so it is
normally visually apparent that scrolling is possible.
Also, all other suggested changes that have not been implemented was mostly because of project time
constraints (and scope) but were all noteworthy remarks. Essentially, refining core wiki functionality had
priority over the gamification additions.
7.5.2 Suggested changes implemented
Create page
The place page option has now been removed as users never felt it was necessary.
Created page
Every wiki page created or even viewed is now displayed in a less cluttered form (without the
points gauge) (see figure 28).
General
Points gauge now only occurs in the home screen (as suggested by one user) as most users did
mention that it was causing clutter in other screens and distracting from the screen‟s purpose.
The leaderboard is now only accessible from the home page (and not the game profile as well)
since this was suggested by a user. Indeed, this does remove the ambiguity of having two of the
same leaderboards in two different places.
74
Smaller icons and images have now been used across the system, as users did request this and it
speeds up their navigating of the gamified wiki (see figure 28).
7.5.3 General system changes made
In addition, to changes suggested by users, there has been some new features (most of which derived even
at paper prototype 2) and refinements added to the system, in order to finalize it.
Added
The group functionality (even though limited) has now been added to the system (allows users to
compete in groups). It allows a user to create/join/leave a group and to view the leaderboards by
groups. Importantly, the group feature was also described in the gamification project which improved
University pass rates(D. W. Bustard, M. M. Black, et al. 2011) and users have approved of it during
the paper prototype 2 evaluation.
In addition, the leaderboard on the home page can now be filtered by groups or individuals. Also, the
leaderboard supports a drill-down functionality which is once again inspired by the successful
gamification that took place at Ulster university rates(D. W. Bustard, M. M. Black, et al. 2011).
Essentially, this meant when clicking on an individual in the leaderboard, one could see how they
acquired their points (in a rewards list screen). However, this is a static screen which always shows
the same rewards, as it has not been demanding by users that much.
Removed
The menu-disabling functionality, is now removed even though it was a form of error prevention.
Ironically, it was causing more errors (due to inappropriate disabling of screens) than it was
preventing.
All the search bars including the ones found in screens: view wiki pages, place section, create group
screen; will not be implemented. These were merely optional and therefore were forfeited eventually
due to time constraints. However, in terms of scale, a search would be necessary if the wiki has
thousands of pages.
7.6 Conclusion Although, this chapter is mainly about the final implementation, it begins by describing the initial
feasibility test. In terms of implementation, the technology of the initial feasibility study (such as
jQueryMobile) was the same technology used in the final system. Also, it is worth noting that the
feedback from the initial feasibility study was applied in future prototype iteration.
Thereafter, the final implementation is described, which supersedes the horizontal software prototype.
This is a top-down description, starting with the system architecture, and later going on to methodology
where issues such as maintainability, portability and the systems limitations are addressed, till the actual
application screenshots are shown. Indeed, the screenshots do show that the final system takes up far less
screen space than the horizontal prototypes (at least for these screens). Then, the suggested changes
implemented and not implemented were reported on. There were a few changes added to the system, but
for the most part, many of the suggestions were not implemented in this iteration as there were probably
gamification elements which are more optional than core wiki functionality.
75
8 Experimentation Design and Results
8.1 Introduction
In the past iterations of the project, formative evaluations were conducted to improve system usability
iteratively(Gediga et al. 1999). Now, that the system is in a final stage, it can be evaluated using a
summative evaluation, as this is a typical usability assessment of a final design(Nielsen 1993) and
determines if system has meet its objectives(Hamilton & Chervany 1981). The summative evaluation is
conducted using experimental evaluation(Hezel Associates 2010) because it aids in finding the objective
truth i.e. truth that is independent of any bias(Jones & Marsden 2005, p.210). This evaluation was
conducted over two days and with 10 users.
The goal of the experiment is to compare the usability of the current mobile Vula and gamified wiki. This
is elaborated in the first section (hypotheses), thereafter; the user choice, experimental design, control
procedures and experiment tasks are discussed, in turn.
8.2 Method
8.2.1 Hypotheses
Setting the hypotheses is the first step to an experimental evaluation, however in order to do this, the
dependent and independent variables need to be defined(Jones & Marsden 2005, p.238). These variables
are tabulated below (table 10):
Table 10: Dependent and independent variables
Dependent Variables Independent Variables
*Number of clicks(measured relatively as
screen transitions)
System choice(the Vula and gamified mobile
wikis)
*Task completion time(measured absolutely
in seconds)
Number of errors(in completing tasks)
*These variables measure the time taken to complete a tasks either relatively (comparative measure of
both systems) or absolutely(precise measurement to the second)(Jones & Marsden 2005, p.238).
As mentioned before, the research question investigates whether a gamifiedVula wiki on mobile is more
usable than the current Vula wiki on mobile.Usability in this context is measured by effectiveness
(number of error rates) and efficiency (number of clicks, task completion time) and to a lesser extent user
satisfaction(Nigel Bevan 1995). Therefore, the research question can be broken down into three separate
hypotheses (where mobile abbreviates mobile phone):
76
Hypotheses
2. A gamifiedVula wiki on mobile is more usable than the current Vula wiki on mobile.
2.1 For all tasks, agamifiedVula wiki on mobile requires fewer key presses to navigate than the
current Vula wiki on mobile.
2.2 For all tasks, agamifiedVula wiki on mobile requires less time to navigate than the current Vula
wiki on mobile.
2.3 For all tasks, agamifiedVula wiki on mobile is less error prone than the current Vula wiki on
mobile.
Null Hypotheses
1. A gamifiedVula wiki on mobile has the same usability as the current Vula wiki on mobile.
1.1 For all tasks, agamifiedVula wiki on mobile has the same navigation key press as the current
Vula wiki on mobile.
1.2 For all tasks, agamifiedVula wiki on mobile has the same navigation time as the current Vula
wiki on mobile.
1.3 For all tasks, agamifiedVula wiki on mobile is as error prone Vula wiki on mobile.
Limitations:
The effect of the gamification on the new wiki(part of user satisfaction) could not be evaluated fully as
this requires a study that would run over some time(Fitz-walter, Tjondronegoro & Wyeth 2011b), with
each user needing a smart-phone at home and furthermore, the results of such a study is very subjective(or
user opinion based)(Fitz-walter, Tjondronegoro & Wyeth 2011b).Indeed, user satisfaction could not be
analyzed comprehensively due to the lack of availability of professional mobile usability
questionnaires(Jones & Marsden 2005, p.205). In addition, due to time constraints the extra evaluation of
the tullis interface display complexities(Tullis 1984) of both wikis could not be carried out.
8.2.2 Users chosen
Firstly, the appropriate ethical clearance from the Science Faculty Ethics clearance and access to UCT
students (from UCT student affairs) has been obtained.Also, having one independent variables (system
type), and two possible options (Vula and gamified mobile wikis), it meant that 20 users were needed but
since a within-group study is used only 10 users were chosen for this evaluation (which is of great
financial relief).Also, there was an equal number of novices (users who never used the neither the Vula
nor gamified wiki before) and experts (users who have used the Vula and gamifiedwiki). This is because
some students have used the Vula wiki before and it alleviates that learning effect, since the gamified wiki
has only been used by a few. Likewise, a two-system mobile study included half novices and
experts(Jones & Marsden 2005, p.239)however the effect of including novice or expert users in studies is
still being debated (Nielsen 2000).
This was achieved through pre-experiment conversations with the users(Jones & Marsden 2005, p.239).
77
The users consisted of undergraduate computer science students of all years were chosen as they are the
end-users for which the system is developed. Importantly, the students were chosen randomly through
tutorial announcements, chatroom posts and hand-picking users (who met end-user criteria) in computer
laboratories.In addition, users of various ethnic groups and both genders were present.
8.2.3 Experimental Design
The final evaluation took place in controlled settings which where various University of Cape Town
computer science laboratories, quiet classrooms and a residence room. Participants conducted evaluations
for approximately 40 minutes each, over 2 days (3 in day 1 and 7 in day 2). Indeed, it worth noting that a
pilot experiment was not undertaken due to lack of time but all the experiments ran successfully,
nevertheless. Furthermore, many of the core tasks of the final experiment are repeated from the previous
formative evaluation such as comparing the number of clicks and errors in performing tasks in both wikis.
In terms of number of final experiments, since there is one independent variable (system choice) and two
options (the current and gamified mobile Vula wiki), there are two conditions resulting in two
experiments (one per system) in total. The first part of the experiment compares the usability
(effectiveness and efficiency) of the current and gamified mobile Vula wiki (the project‟s main research
question) in terms of number of clicks (i.e. screen transitions) and errors to perform given tasks. Also,
although half of the users chosen have experience in both wikis, this data will be recorded as normal for
both groups as a similar study did not distinguish these groups either in data collection(Jones & Marsden
2005, p.239). Now, the actual procedure is to be described:
Initially, students were first assured that their names would not be published (ethical anonymity) and also
asked whether they agreed for a(optional) tape recording. Although, tape recordings can be used to help
transcribe information (Jones & Marsden 2005, p.155), they were used in this case as evidence that the
evaluations actually took place. Also, the students were also assured of their R30 payment after the
experiment but asked to remember to sign the consent form(Chin 2001) after evaluation (to also indicate
they have received payment). Each subject is given a brief explanation(Chin 2001) of how wiki‟s work,
but no demonstration as to how to navigate (as this is being evaluated).Thereafter, students were also told
that the tasks would take roughly 2 minutes (although longer times were recorded) and the experiment
began. Then, the student was handed the Samsung Galaxy Ace (sponsored project phone) whilst the
evaluator recorded data on a laptop using a pre-defined coding sheet/evaluation grid(Chin 2001).
Indeed, automatic software logging(Guzdial et al. 1994) could also be used to record data but did not
seem worth the effort as the formative evaluation succeeded without it.Each of the 10 students performed
five tasks (per system) in sequence and was given limited guidelines where truly necessary but these were
noted as errors in tasks completion. Furthermore, to avoid experiment bias, the order in which each of
these tasks were performed was inverted (Jones & Marsden 2005, p.239) after each experiment and even
the order of which system the user evaluated first(Jones & Marsden 2005, p.213).The later technique is
known as a within-group study, and is chosen since it requires fewer users than a between-group
study(Jones & Marsden 2005, p.213) . A between-group study, is one in which double the amount of
users are needed as each user tests only one system and alleviates learning effect in that way (Jones &
Marsden 2005, p.213)
78
The wiki and gamification databases for the gamified wiki were reset for the purposes of the experiment.
However, since it was not possible to hide old wiki pages in the traditional Vula wiki, the pages needed
for this experiment were indexed for fast access on both wikis. This ensured consistent data access across
both wikis; however the data between iterations differs as users applied new edits in their iterations.
After completing the tasks, the second part of the experiment assessed user satisfaction and gamification
influence (to a small extent) by a post-interview session(Jones & Marsden 2005, p.241) with users
(lasting about 10 minutes). Rather than having use a disruptive think aloud evaluation to gather users
thoughts during the evaluation, the post-experiment interview solicits user opinions (about the systems) in
an unobtrusive way(Jones & Marsden 2005, p.240). Furthermore, interviews are known to have the best
value when used in conjunction with a laboratory based evaluation hence they are used here as well(Jones
& Marsden 2005, p.205). The interview questions(see Appendix 11.6), inspired by a past thesis of a
previous UCT Honours student (Palser 2004) and a usability website (Hezel Associates 2010), probed
user satisfaction areas such as: user anticipation, user experience and future recommendations. Finally,
the participant was then asked to sign the consent register, then thanked for their efforts and paid R30 for
their participation. The register includes their email addresses and contact numbers (optional), so it serves
as quality control that the experiments did take place and for the department to later refund the researcher
for user evaluations.
8.2.4 Control procedures
The control procedures are discussed throughout this chapter but the following relate to the statistical
Wilcoxin signed-rank test, which was used to evaluate the various null-hypotheses:
Normality:
This condition is not met as the data is too small(10 users) but normality is not a pre-requisite with the
Wilcoxin signed-rank test (McCluskey & Lalkhen 2007).
Random selection:
Students were chosen randomly(Chin 2001) through tutorial announcements, chartroom posts and hand-
picking users(who met end-user criteria) in computer laboratories. This helps alleviate the effect of the
experimenter biasing the experiment by words, appearance, attitude, etc(Chin 2001).
Independence:
The occurrence of a measurement in one variable should not change the occurrence in another variable.
Therefore, no user was allowed to invite their friends(Dr. Hun Myoung Park 2009).
8.2.5 Tasks
The tasks chosen are representative of the real world usage of the system(Jones & Marsden 2005, p.212)
and furthermore evaluates the research question of whether gamification improves the usability of the
Vula wiki. Each student completed 5(of the same tasks) for each system, 5 tasks for each
condition/system (Jones & Marsden 2005, p.239) . Indeed, the tasks are the same ,instead of different like
79
in this study (Jones & Marsden 2005, p.239), so a simple comparison can be made between the 2 wikis .
Furthermore, possible learning effects, which is students performing better/worse on one system after
using another (Jones & Marsden 2005, p.213), was eliminated in two ways. The first was to invert the
order of tasks after every run (Jones & Marsden 2005, p.239) and also to use a within-group test (Jones
& Marsden 2005, p.212) where each successive participant starts with a different system. Also, the tasks
do not bias either system (as much as possible) (Jones & Marsden 2005, p.212) for instance: the gamified
wiki is optimized for adding article paragraphs(called sections) anywhere in the page thus this is not
tested explicitly. However, attempts to avoid wording bias of task(Jones & Marsden 2005, p.239), which
is the naming of task that make user no exactly how to perform an action, could not be achieved for all
tasks. Indeed, it is easy to provide synonyms (Jones & Marsden 2005, p.240) for simple actions like
“create page” to “make a page” but activities such as tutorial sign up, it was too difficult to find
alternatives. Even, in the case of “make a page”, this biases the gamified wiki as it has a “+page” menu
icon but the Vula wiki also benefits from the “edit page” tasks as it has an “edit icon”. Overall, it is hoped
that these imbalances cancelled each other off. Also, the tasks were the same for novice users and expert
users as a similar study used the same approach(Jones & Marsden 2005, p.239) .
As above mentioned, the tasks were carried out on the same mobile galaxy ace ensuring consistent
hardware and user experience(Jones & Marsden 2005, p.240).The tasks were mainly core wiki navigation
tasks as that makes usability easier to analyze whereas determining the impact of gamifcation would need
a more prolonged study(Fitz-walter, Tjondronegoro & Wyeth 2011b). Lastly, to follow is the actual tasks
list (with a description of each in the appendix 11.4):
Make a page?
Edit a page?
Add section/Add a paragraph with heading and content?
Find out when the first operating system was made?
Sign up to a tutorial?
Finally, it also worth noting that the operating system article(Wikipedia 2012a) is adapted from
Wikipedia(Wikipedia 2012b), and the tutorial sign up is adapted from the CS3003S(a UCT course) Vula
wiki.
8.3 Findings and Results Given, that the final experimental/summative evaluation is complete, this section analyzes, discusses
results to be concluded on in the next chapter. Firstly, the quantitative results (like average clicks) of the
experiments are presented, analyzed and discussed including the statistical significance of the data.
Thereafter, the qualitative results in terms of user opinions are also discussed to be followed by problems
found in the gamified wiki (both in scope and out) as well as solutions where applicable. Finally, in the
next chapter, conclusions are drawn out of these results in relation to the projects hypotheses questions as
well as the summary of the work and suggestions for future research.
80
8.3.1 Results analysis and discussion
The three null-hypotheses(see next paragraph) are tested using a Wilcoxon signed-rank test, as the
distribution of data is non-normal and the sample is small/”participants less than 30”(H. J. Arnold 1965).
The null-hypotheses are tested task by task by finding the difference in task performance between the two
systems for each task, as supposed to a test for the entire dataset because this provides the statistical
significance for each task. Also, it is worth noting that the statistical test results are subject to the
limitations described at the end of this section. Next, is just a reminder of the null hypotheses and then
they will be evaluated in turn.
Null Hypotheses
1. A gamifiedVula wiki on mobile has the same usability as the current Vula wiki on mobile.
1.1 For all tasks, agamifiedVula wiki on mobile has the same navigation key presses as the current
Vula wiki on mobile
1.2 For all tasks, agamifiedVula wiki on mobile has the same navigation time as the current Vula
wiki on mobile
1.3 For all tasks, agamifiedVula wiki on mobile is as error prone Vula wiki on mobile
Average clicks test
The following graph below compares the average clicks per tasks of the mobile Vula wiki and the mobile
gamified wiki.
Figure 31: Average Clicks per Task graph
0
0.5
1
1.5
2
2.5
3
3.5
1 2 3 4 5
Ave
rage
Clic
ks
Tasks
Average Clicks per Task
Vula
Gamified
81
The average clicks/”screen transitions” (for all tasks) in current Vula wiki is found to be 2.34 clicks
compared with 2.51 clicks in the gamifed wiki. Indeed, from the figure above, the gamified wiki clearly
has more clicks for task 1 and 5, but the rest are about equal. Now, the statistical analysis is to follow.
A Wilcoxon signed-rank test is made on each task (see appendix 11.8) and there was no statistically
significant results (at a 5% significance level) for any task as this sample size was too small generally
(McCluskey & Lalkhen 2007). Indeed, in one of the tests there was only 1 paired observation whilst in
others there were 9. Therefore, the null hypothesis is not rejected at a 5% significance level (or 95%
confidence level) which means (given the experimental limitations) that there is no difference in the
number of clicks per task in the mobile Vula wiki compared to the mobile gamified wiki (with 95%
certainty).
Average task completion time test
The following graph below compares the average task completion time of the mobile Vula wiki and the
mobile gamified wiki.
Figure 32: Average Task Completion Time graph
The average task time in current Vula wiki is found to be 131.56 seconds compared with 101.79 seconds
in the gamifed wiki. Indeed, this is quite clear from the above graph, as the Vula wiki has longer times for
every tasks and especially task of create page, which most users struggled with on the Vula wiki (as only
3 successful). Now, the statistical analysis is to follow.
Surprisingly, after conducting a Wilcoxon signed test on all tasks; the gamified wiki was found to be
statistically significantly faster for one task(but at a 10% significance level). This task was to edit a page
and probably showed this significance because it had 9 paired observations and difference of
0
20
40
60
80
100
120
140
160
1 2 3 4 5
Ave
rage
Tim
e(in
sec
on
ds)
Tasks
Average Task Completion Time
Vula
Gamified
82
times(between systems) was 29 seconds or it was a random occurrence(Rukhin et al. 2001). However, all
other tasks did not show any significance (even at 10% significance level). Therefore, the null hypothesis
is not rejected at a 5% significance level (or 95% confidence level) which means (given the experiment
limitations) that there is no difference between the average task completion time in the mobile Vula wiki
and the mobile gamified Wiki.
Average errors test
The following graph below compares the average number of errors per task of the mobile Vula wiki and
the mobile gamified wiki.
Figure 33: Average Errors per Task graph
The average number of errors per task (i.e. a wrong action by user) in current Vula wiki is found to be
131.56 seconds compared with 101.79 seconds in the gamifed wiki. Indeed, this is quite clear from the
above graph, as the Vula wiki has longer times for every tasks and especially task of create page, which
most users struggled with on the Vula wiki (as only 3 successful). Now, the statistical analysis is to
follow.
Notably, this test was the only one with all observations present. Once again, after conducting a Wilcoxon
signed test on all tasks; the gamfied wiki had statistically significant lower average error but for only 1
task (at a 5% significance level). This task was to create a page and probably showed this significance
because it had all 10 paired observations and a difference in errors for half of the task or it is a random
occurrence(Rukhin et al. 2001). However, no other tasks showed statistical significance despite having all
paired observations because the observations were not very different. Therefore, the null hypothesis is not
rejected at a 5% significance level (or 95% confidence level) which means (given the experiment
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1 2 3 4 5
Ave
rage
Err
ors
Tasks
Average Number of Errors per Task
Vula
Gamified
83
limitations) that there is no difference between the number of errors in the mobile Vula wiki and the
mobile gamified wiki.
Limitations
Sometimes non-parametric evaluations like the Wilcoxon signed-rank test(H. J. Arnold 1965), gets p-
value wrong for very small sizes(McCluskey& Lalkhen 2007).
Missing data entries(i.e. tasks users could not complete) were rectified using pairwise deletion, hence
weakening observation quality(Croninger& Douglas 2005). Indeed, this could have been avoided by
piloting this experiment to see if users could complete most tasks.
Standard errors could not be computed since errors change with the sample size due to pairwise
deletion(McCluskey& Lalkhen 2007).
Flaws
The experiment should have been carried out with more participants and experiment should have
been more carefully designed to avoid missing data(i.e. users not completing tasks)(Fleming 2011).
Indeed, it is known that larger samples increase chance of significance due to sensitivity of Wilcoxin
signed-rank test(as it is non-parametric) (McCluskey & Lalkhen 2007). Further, in the future more
evaluators could be useful to record data more accurately, as there is two cases where task time is
recorded for a task but the number of clicks not.
8.3.2 Usability Feedback
Although, the statistics highlighted the main results, the additional usability feedback also provides
valuable user insights. Indeed, one needs to be aware of the Hawthorne effect(Adair 1984) , which is
about users trying to please the experimenter with their feedback but even so their opinions are always
valuable, being that they are the target-group of the gamified wiki.
Ease of Use
When asked about the overall ease of use both applications as mentioned above, the majority of users said
that the gamification wiki was much simpler to use. In fact, the average user rating (of 10 users) for the
mobile Vula wiki is 1.5 and the mobile gamified wiki is 4, using a 5-point Likert scale shown in the
interview questions(appendix 11.6, question 6). Some comments included, “It is very convenient, so easy
to access. The pages look so much better”. Though, another mentioned that: “It is simple but some things
are still complex”. When probed further about this, the person mentioned that: “Adding sections, I did not
quite understand”. Users had the following to say about the Vula wiki, “Not as easy as the gamified one,
no specific buttons to do tasks”. Afterwards, the user elaborated that there are no specific buttons to create
a page on the current Vula wiki and yet another user mentioned that the edit button on the current Vula
wiki is too ambiguous as it does not say what is being edited. Nevertheless, the gamifiedVula wiki was
not without its flaws, as most users did express irritation at the delayed scroll of the menu and more of
these are to follow in the forthcoming section. Before that, it is also worth noting that positive user
84
feedback as a result of gamification was also reported for the gamified orientation application at
Queensland University(Fitz-walter, Tjondronegoro & Wyeth 2011b).
Likes/Dislikes
One user mentioned that, “The current Vula wiki loads faster than the new one and has nice icons”. Then,
about the gamified wiki, the user mentioned knowing exactly what each button did and it was much
easier to use, such as the use of a textbox to enter content (section)”. Indeed, in the Vula wiki, the
entire page is edited at a time whereas in the gamified wiki only sections are edited at a time, like
Wikipedia(Wikipedia 2012b). Furthermore, most users did not appreciate the complicated mark-up
(when editing) in the Vula wiki, one describes it as: “Complicated random coding in the middle of the
text box”.
Also, one of the main likes of the gamified wiki was the fact that a page is displayed immediately after it
is created (instant feedback). Whereas, on the current wiki, the user has to search right down the
current page (or parent page) and look for the link of the newly created page.
Then, there is the simplicity of creating new pages in the gamifiedVula wiki whereas only 1 user was able
to create a page in the current Vula wiki. This is echoed in this user remarked:” I like the fact that you can
actually create pages (in the gamified wiki).
However, the search feature in the Vula wiki was appreciated by one user even though it was used by
about half. This user remarks: “The current wiki has a search button. If the gamified wiki had this too, it
would be quicker.”
In terms of screen clutter, it has been mentioned that: “the gamified wiki is like the old Vula wiki without
all the junk (referring to screen). “ Furthermore, a user added, that:” the gamified wiki was spaced and
had smaller info on each page, with better icons such as „+page‟.” However, the majority of users agreed
that some icons (i.e. badges and game content) were unnecessarily large and as one user says: “some of
the icons were larger than necessary, a bit too informal.”
Confusing Terminology
The notion of sections seemed to be the most confusing terminology for most users in the gamified
system. As quoted above, “Create sections, I do not quite understand”. Although, the users did show their
appreciation when they realized how to use it but it was simply not a word the identified with
immediately. Also, a user suggested that the shortening of words such as “+sec” which abbreviates
“+section” in the menu, should be replaced with the full names. However, a mobile phone‟s screen width
is too small, to implement this for 5 menu items. One user mentions that the “game” menu icon was not
informative enough. Although, this occurred since they glimpse over the introductory tutorial.
Finding Game Elements
All the users said the “game” icon drew their attention to the gamification element of the gamified wiki.
However, the introductory tutorial screen is meant to draw their attention to the gamification component
yet many skimmed through it. Although, there is another user who mentions: “There is a speedometer,
85
which shows your points. Yes, it did add value since it keeps you interested.” However, the majority of
users did not notice the game elements, mostly since the experiment time was too short to show them, as
one user says: “Did not really think that it was a game but maybe need to use it more.” Yet another says:
“It was not that fun but not boring (either)”. Also, one user remarked, “It (the wiki) was encouraging, it
made me go back and put my whole life story into it. The little leaderboard made me competitive. ” In
fact, the leaderboard is intended to raise user competitiveness with each other(Burke & Hiltbrand 2011).
Interesting Observations
One user who struggled to find create page in the current Vula wiki, edited a page called „mypage‟
(created by another user), in order to create their own page. Even though this occurred in the old wiki
interface, it just shows that due to the anyone-can-edit nature of a wiki(Cunningham 2012), unforeseen
consequences can occur. Similarly, a user went to the join groups screen when instructed to sign up for a
tutorial and expected to find a wiki page called tutorial sign up and edit that. Indeed, the naming of the
wiki page conflicted with an element already on the system, potentially as a back-end future work a
naming algorithm could check for these conflicts. Also, the majority of users never realized that the like
button was clickable as it looked just like an image. One user suggested that it should look more like a
button and have text underneath saying “like/unlike”, which is exactly the Facebook layout. Indeed, this
can be seen as a problem of affordance(Jones & Marsden 2005, p.67) , as an image does not naturally
invite a user to click it .
Recommendations
“Just add a search and that is all”, is one user‟s remark. Another student has the following to say about the
home screen, “The view wiki button should be grouped separately from the gamification elements”.
Given, that this does tie in with logical grouping in mobile phones(Shneiderman& Plaisant 2005, p.498)
but this is not necessary as no other user saw this as a problem. Also, a student found that there is no need
to show the user all the available avatar level pictures(even if not unlocked) as it means removes the
need/drive(Antin& Churchill 2011) to unlock them as this is default. In addition, the gamification icons
(in wiki articles) such as the contributors badge rating, were found to be too space consuming and should
be reduced in size as this comment suggests, “The legendary picture is way too big and it takes up a
whole screen.”
8.3.3 Problems with the application
Adding page search feature
“Just add a search and that is all”, is one user‟s remark. By this, the user was referring to a page search
feature which may be added later (since most users did not point this out).
Delayed scrolling of menu when page scrolls
This was the most noticed problem by users, illustrated by this quote, “I don‟t like the menu bar going up
and down”. As mentioned above, the menu bar does not stay at the bottom of the page during scrolling
86
but returns there once the scrolling stops. Essentially, the menu bar needs to always be at the bottom of
the screen.
Edit/Create Section Screen:
The cancel button in these screens does not go back to the previous (wiki) page but rather the home
page. Admittedly, this is a simple flaw that was overlooked by the researcher and will be fixed.
The cancel button confirmation message is often unnoticed because it is displayed beneath the cancel
button (needing the user to scroll to see it) whereas it would be more visible above the cancel button.
All pages screen:
The buttons are too small in this screen as one user puts it, “Every now and again the buttons are too
small. Maybe put some space between them. ” When probed further, the user was mainly referring to the
view wiki screen and their suggestion will be considered.
Re-grouping home screen:
“The view wiki button (in home screen) should be grouped separately from the gamification elements (as
it does not relate to them)”. Given, that this does tie in with logical grouping in mobile
phones(Shneiderman& Plaisant 2005, p.141) but this is not necessary as no other user saw this as a
problem .
Game profile screen:
Also, it was discovered that there is no need to show a user all the available avatar level pictures(even if
not unlocked) as it means removes the need/drive(Antin& Churchill 2011a) to unlock them as this is
default.
Display of wiki pages:
In addition, the gamification icons (in wiki articles) such as the contributors badge rating, were found to
be too space consuming and should be reduced in size as this comment suggests, “The legendary picture
is way too big and it takes up a whole screen.”
8.3.4 Problems beyond system scope
Firstly, there has been a suggestion for a chatroom, “No chatroom, if there is someone online, we can
chat.” In fact, this would not be a problem if this wiki (i.e. the gamifed one) is actually embedded in a
Vula course site because the standard chatroom tool would be available to enable this. However, this is
mainly future work for the back-end component, to integrate the gamified wiki into the current Vula
interface. Also, there is a need for a „naming‟ algorithm to ensure that wiki pages created do not conflict
with names of elements currently in the system, maybe of benefit. Although, this is a back-end extension,
it was conceived through this evaluation and will prevent problems(like previously mentioned) of users
choosing a system function because it is on the home screen(like join group) instead of finding a wiki
page(tutorial sign up) , which is what they originally intended.
87
9 Conclusion
The aim of the project is to determine if gamification can improve the usability of the Vula (Sakai) wiki
on mobile phones because the current Vula wiki tool is under-utilized. Since the wiki can be a valuable
educational tool as a gamified system (for instance) at the University of Ulster even led to improved pass
rates, it was decided thatgamificiation should be used to make the wiki more easy to use and engaging.
In terms of contributions, as far as we could ascertain this is the first research done on the gamification of
a wiki and is also unique in trying to use gamification to improve mobile usability. This was done by
building a system using the user-centered design approachproducing and evaluating two low-fidelity
paper prototypes and two high-fidelity prototypes. In addition to user-suggested improvements that
resulted from these iterations, the game rules were reduced over time and are influenced largely by the
GameFlow criteria for player enjoyment in games.
The final implementation, which uses jQuery Mobile, incorporates feedback from the 4 design iterations,
integrates with the back-end and includes gamification functionality discovered to be successful for
education at the University of Ulster.
Findings
Did gamification improve the usability of the Vula wiki? Not statistically, the results showed that there
was no statistically significant difference between the mobile Vula wiki and mobile gamified wiki (for all
null-hypotheses) therefore the usability is the same and has not been improved. However, the gamified
wiki was statistically significantly faster for 1(out of 5) tasks and had statistically significantly less errors
for 1(out of 5) tasks. This could have been random occurrence or signs of a minor improvement in
usability.
Limitations of the experiment such as a small sample size(10 participants) substantially lowered the
chance of detecting statistically significant results and this in turn blurred out whether the wiki truly
improved in usability or not. Furthermore, non-parametric tests sometimes produce the wrong p-value for
small sizes hence inaccurate results. Also, the quality of the observations is also weakened due to missing
data entries.
As far as user opinion went, the gamified wiki is far easier to use as they gave it an average rating of 4/5
versus 1.5/5 for the Vula wiki. However, one must note the Hawthorne effect of users trying to please the
experiment facilitator. Similarly, previous a gamification of a university orientation application at
Queensland University also yielded positive user feedback.Also, a limitation of this project‟s post-
interview questions is that they are not scientifically validated so the reliability of results is uncertain.
Overall, there has been no (significant) improvement in the usability of the Vula wiki through
gamification, except for the one task with statistically significant improvement (at a 5% significance
88
level) and another (at a 10% significance level). However, this is the first time gamification has been
introduced to the Vula wiki and has been met by positive feedback (although honesty is not assured).
We believe this project to be a first step in tackling an important problem because it seems not many
students do use the Vula wiki based on this research. However, the experiment could have been improved
by including a greater number of participants and to avoid missing data by first running a pilot experiment
to see if users could complete all the tasks. Indeed, this could possibly lead to more statistically
significant results.
On the other hand, the user-centered design methodology certainly helped understand users‟ needs better
to produce a system they appreciated.
This experience has been a humbling one, which shows that user feedback always changes and one needs
to adapt quickly to it. More importantly, it has shown that research takes time, so that is why some of it
has to be left to future work which is described next.
9.1 Future work The most immediate suggestion is to replicate the experiments with a greater number of participants
(above 30). Possibly, further research can also examine the reasons for better or worse usability when
gamification has been applied.
In terms of gamification functionality, what is of immediate need is a notification system which will alert
a user when they have unlocked a reward (providing constant feedback). In part, the message box icon on
the top left of most screens was intended to highlight this. Some modifications to the game rules would be
desirable, such as only awarding points for articles of sufficient length. Secondly, a question-and-answer
system like StackOverflow can be added to the wiki, as well as other features in like having a different set
of game rules and badges for groups.
The contribution of each user could be highlighted in a different color but on a mobile phone usability
color guidelines must be adhered to. Also, the search functionality for wiki pages can be added as well as
an interactive tree-map display that shows article popularity and some visual aid could be used to show
that scrolling is possible in a page.
Lastly, in order to deploy the system there has to be extensive testing of robustness and scalability.
Furthermore, some back-end work needs to be made to integrate the system into Vula(just like the current
Vula wiki). Hopefully, this research will generate more investigations into the use of wiki gamification in
education.
89
10 References
Adair, J.G., 1984. The Hawthorne effect: A reconsideration of the methodological artifact. Journal of
applied psychology, 69(2), p.334.
Adobe, 2012. HTML editor software, web-design software | Adobe Dreamweaver CS6. Available at: http://www.adobe.com/africa/products/dreamweaver.html?kw=p&sdid=JTAIZ&skwcid=TC|22636|
dreamweaver||S|e|25259413435 [Accessed October 26, 2012].
Antin, J. & Churchill, E., 2011a. Badges in Social Media : A Social Psychological Perspective. Human
Factors, Human Fact, pp.1-4. Available at: http://research.yahoo.com/node/3469.
Antin, J. & Churchill, E., 2011b. Badges in Social Media : A Social Psychological Perspective. Human
Factors, Human Fact, pp.1-4. Available at: http://research.yahoo.com/node/3469.
Arnold, H.J., 1965. Small sample power of the one sample Wilcoxon test for non-normal shift
alternatives. The Annals of Mathematical Statistics, pp.1767-1778.
Arnold, R., 2010. Giving users what they don‟t know they want. Available at:
http://tabulacrypticum.wordpress.com/2010/07/23/giving-users-what-they-dont-know-they-want/
[Accessed October 1, 2012].
Aumüller, D., 2005. SHAWN: Structure helps a wiki navigate. In In Proceedings of the BTW-Workshop
WebDB Meets IR. Karlsruhe, Germany. Available at: http://dbs.uni-
leipzig.de/file/aumueller05shawn.pdf [Accessed July 30, 2012].
Berman, P.S., 2012. Vula wiki underuse.
Bevan, N & Azuma, M., 1997. Quality in Use: Incorporating Human Factors into the Software
Engineering Lifecycle. In Proceedings of the 3rd International Software Engineering Standards
Symposium (ISESS ’97). Washington, DC, USA: IEEE Computer Society, p. 169--. Available at:
http://dl.acm.org/citation.cfm?id=850975.854938.
Bevan, Nigel, 1995. Measuring usability as quality of use. Software Quality Journal, 150.
Bevan, Nigel, 2001. International standards for HCI and usability. International Journal of Himan-Computer Studies, 55(4), pp.533-552. Available at:
http://linkinghub.elsevier.com/retrieve/pii/S1071581901904835 [Accessed July 30, 2012].
Beydeda, S. & Gruhn, V., 2001. Integrating White- and Black-Box Techniques for Class-Level
Regression Testing. In Proceedings of the 25th International Computer Software and Applications Conference on Invigorating Software Development. Washington, DC, USA: IEEE Computer
Society, pp. 357-362. Available at: http://dl.acm.org/citation.cfm?id=645983.675252.
Di Bitonto, P., Roselli, T. & Rossano, V., 2009. Formative evaluation of a didactic software for acquiring
problem solving abilities using Prolog. In Proceedings of the 8th International Conference on
Interaction Design and Children. New York, NY, USA: ACM, pp. 154-157. Available at:
http://doi.acm.org/10.1145/1551788.1551815.
90
Buchanan, G. et al., 2001. Improving mobile internet usability. In Proceedings of the 10th international
conference on World Wide Web. New York, NY, USA: ACM, pp. 673-680. Available at:
http://doi.acm.org/10.1145/371920.372181.
Burke, M. & Hiltbrand, T., 2011. How Gamification Will Change Business Intelligence. Business
Intelligence Journal, 16(2), pp.8-16. Available at:
http://proxy2.hec.ca/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=bth&AN=61
965733&lang=fr&site=bsi-live.
Bustard, D., Black, M. & Charles, T., 2011. GEL: A generic tool for game-enhanced learning. , Belfast, UK, iNEER. Available at:
http://ineerweb.osanet.cz/Events/ICEE2011/papers/icee2011_submission_105.doc [Accessed May
14, 2012].
Bustard, D.W., Black, M.M., et al., 2011. GEL : A Generic Tool for Game-Enhanced Learning. In
Proceedings of International Conference on Engineering Education, Belfast, UK, iNEER.
Butts, L. & Cockburn, A., 2002. An evaluation of mobile phone text input methods. Aust. Comput. Sci.
Commun., 24(4), pp.55-59. Available at: http://dx.doi.org/10.1145/563997.563993.
Chen, J., 2007. Flow in games (and everything else). Communications of the ACM, 50(4), p.31. Available
at: http://portal.acm.org/citation.cfm?doid=1232743.1232769.
Chin, D.N., 2001. Empirical Evaluation of User Models and User-Adapted Systems. User Modeling and
User-Adapted Interaction, 11(1-2), pp.181-194. Available at:
http://dx.doi.org/10.1023/A:1011127315884.
Clker.com, 2012. Thums Up clip art. Available at: http://www.clker.com/clipart-thums-up.html [Accessed
October 26, 2012].
Croninger, R.G. & Douglas, K.M., 2005. Missing data and institutional research. New directions for
institutional research, 2005(127), pp.33-49.
Crowley, D. & Selvadurai, N., 2009. Foursquare. Available at: https://foursquare.com/ [Accessed May
13, 2012].
Cunningham, W., 2012. Portland pattern repository‟s wiki. Available at: http://www.c2.com/cgi/wiki
[Accessed July 30, 2012].
Deterding, S., Dixon, D., Khaled, R. & Lennart, N., 2011a. From game design elements to gamefulness:
defining gamification. Proceedings of the 15th International Academic MindTrek Conference:
Envisioning Future Media Environments. Tampere, Finland, pp.9-15. Available at:
http://dl.acm.org/citation.cfm?id=2181040 [Accessed April 27, 2012].
Deterding, S., Dixon, D., Khaled, R. & Lennart, N., 2011b. From game design elements to gamefulness: defining gamification. Proceedings of the 15th, pp.9-15. Available at:
http://dl.acm.org/citation.cfm?id=2181040 [Accessed April 27, 2012].
91
Deterding, S., Sicart, M., Nacke, L., O‟Hara, K., et al., 2011c. Gamification. using game-design elements
in non-gaming contexts. Proceedings of the 2011 annual conference extended abstracts on Human factors in computing systems - CHI EA ’11, Vancouver, Canada, p.2425. Available at:
http://portal.acm.org/citation.cfm?doid=1979742.1979575.
Deterding, S., Khaled, R., Nacke, L. & Dixon, D., 2011d. Gamification : Toward a Definition D. Tan &
B. Begole, eds. Design Journal, pp.12-15. Available at: http://gamification-research.org/wp-
content/uploads/2011/04/02-Deterding-Khaled-Nacke-Dixon.pdf.
Deterding, S., Khaled, R., Nacke, L. & Dixon, D., 2011e. Gamification : Toward a Definition D. Tan & B. Begole, eds. Design, pp.12-15. Available at: http://gamification-research.org/wp-
content/uploads/2011/04/02-Deterding-Khaled-Nacke-Dixon.pdf.
Donovan, S.O., 2012. Gamification of the Games Course. Technical Report CS12-04-00, Department of
Computer Science, University of Cape Town., pp.1-8. Available at:
http://pubs.cs.uct.ac.za/archive/00000771/01/Gamification_of_the_Games_Course.pdf.
Evans, R. et al., 2012. C ONSTRUCTION OF A U NIVERSITY N AVIGATIONAL. Center of
Excellence in Remote Sensing Education and Research. Available at:
http://nia.ecsu.edu/ur/1112/teams/database/2012-database-team-research-paper-REVISED-
120410.pdf.
Facebook, 2012. Welcome to Facebook- log in, sign up or find out more. Available at:
http://www.facebook.com/ [Accessed October 26, 2012].
Fischer, F. & Troendle, P., 2003. Using the internet to improve university education: problem-oriented
web-based learning with MUNICS. Interactive Learning, (2), pp.497-498. Available at:
http://www.tandfonline.com/doi/abs/10.1076/ilee.11.3.193.16546 [Accessed May 14, 2012].
Fitz-Walter, Z. & Tjondronegoro, D., 2011. Orientation passport: using gamification to engage university
students. Proceedings of the 23rd Australian Computer-Human Interaction Conference ,Canberra,
pp.122-125. Available at: http://dl.acm.org/citation.cfm?id=2071554 [Accessed May 14, 2012].
Fitz-walter, Z., Tjondronegoro, D. & Wyeth, P., 2011a. Orientation Passport : Using gamification to
engage university students. In Australian Computer-Human Interaction Conference. pp. 122-125.
Fitz-walter, Z., Tjondronegoro, D. & Wyeth, P., 2011b. Orientation Passport : Using gamification to
engage university students. In Australian Computer-Human Interaction Conference. pp. 122-125.
Fleming, T.R., 2011. Addressing missing data in clinical trials. Annals of internal medicine, 154(2),
p.113.
Floyd, C., 1984. A Systematic Look at Prototyping. Springer Verlag, pp.1-17.
Foundation, the jQuery, 2012a. Speedometer | jQuery pluggins. Available at:
http://archive.plugins.jquery.com/project/speedometer.
Foundation, the jQuery, 2012b. jQuery Project. Available at: http://jquery.org/ [Accessed October 26,
2012].
92
Foundation, the jQuery, 2012c. jQueryMobile webisite. Available at: http://www.jquerymobile.com/
[Accessed September 4, 2012].
Foundation, S., 2012. Sakai | collobaration and learning - for educators by educators. Available at:
http://www.sakaiproject.org/ [Accessed October 26, 2012].
Foursquare, 2012. Foursquare Location Based Social Networking. Available at: https://foursquare.com/
[Accessed September 27, 2012].
Gediga, G., Kai-Christoph, H. & Düntsch, I., 1999. The IsoMetrics usability inventory: An
operationalization of ISO 9241-10 supporting summative and formative evaluation of software
systems. Behaviour & Information Technology. Available at:
http://www.tandfonline.com/doi/abs/10.1080/014492999119057.
Godwin-Jones, R., 2011. EMERGING TECHNOLOGIES MOBILE APPS FOR LANGUAGE
LEARNING. Language Learning & Technology, June 2011, Volume 15, Number 2 pp. 2–11.
Available at: http://www.llt.msu.edu/issues/june2011/emerging.pdf.
Gong, J., Tarasewich, P. & Science, I., 2004. GUIDELINES FOR HANDHELD MOBILE DEVICE. In
Proceedings of the 2004 Annual DSI Meeting. Boston, Massachusetts, USA, pp. 3751-3756. Available at: http://www.itu.dk/~jeppeh/Materiale til fag/Konceptudvikling/Guidelines for mobile
interface design.pdf.
Google, 2012. Google Images. Available at: http://www.images.google.com [Accessed October 26,
2012].
Guzdial, M. et al., 1994. Analyzing and Visualizing Log Files: A Computational Science of Usability,
Available at: http://smartech.gatech.edu/jspui/bitstream/1853/3586/1/94-08.pdf.
Hamilton, S. & Chervany, N.L., 1981. Evaluating Information System Effectiveness - Part I: Comparing
Evaluation Approaches. MIS Quarterly, 5(3), pp.pp. 55-69. Available at:
http://www.jstor.org/stable/249291.
Hartson, H.R. et al., 1996. Remote evaluation: the network as an extension of the usability laboratory. In
Proceedings of the SIGCHI conference on Human factors in computing systems: common ground.
New York, NY, USA: ACM, pp. 228-235. Available at: http://doi.acm.org/10.1145/238386.238511.
Hezel Associates, L., 2010. Testing the Efficacy and Impact of a Selected PBS TeacherLine Course,
Syracuse, NY. Available at:
http://www.pbs.org/teacherline/courses/common_documents/research/teacherline_course_efficacy-
hezel.pdf.
Ho, D., 2012. Notepad++ website. Available at: http://notepad-plus-plus.org/ [Accessed October 25,
2012].
Holtzinger, A., Treitler, P. & Slany, W., 2012. Making Apps Useable on Multiple Different Mobile
Platforms: On Interoperability for Business Application Development on Smartphones. IFIP
International Federation for Information Processing. Available at:
http://link.springer.com/chapter/10.1007/978-3-642-32498-7_14?null.
93
Holzinger, A., 2004. Rapid prototyping for a virtual medical campus interface. IEEE , vol.21, no.1, pp.
92- 99. Available at:
http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=1259241&isnumber=28149.
Dr. Hun Myoung Park, 2009. Comparing Group Means: T-tests and One-way ANOVA Using Stata, SAS,
R, and SPSS. Available at: http://www.indiana.edu/~statmath/stat/all/ttest/ttest.pdf.
Immunopedia.org, 2010. Immunopedia website. Available at:
http://www.immunopaedia.org.za/index.php?id=4 [Accessed September 26, 2012].
Ingber, L., 2012. Columnar EEG magnetic influences on molecular development of short-term memory S.
F. P. G. Kalivas, ed., Hauppauge, NY: Nova. Available at:
http://www.ingber.com/smni11_stm_scales.pdf.
Iron Realms Entertainment, 2012. A list of experience level names. Available at:
/game/helpview/lusternia/a-list-of-experience-level-names [Accessed October 26, 2012].
Jenson, S., 2002. The Simplicity Shift, Cambridge,England.: Cambridge University Press 2002.
Jones, M. & Marsden, G., 2005. INTERACTION Edition, 1., Wiley.
Jong, T.D., Specht, M. & Koper, R., 2008. A reference model for mobile social software for learning.
International Journal of Continuing Engineering Education and Life-Long Learning, 18(1), p.118.
Available at: http://www.inderscience.com/link.php?id=16079.
Koenemann-Belliveau, J. et al., 1994. Comparative usability evaluation: critical incidents and critical threads. In Proceedings of the SIGCHI conference on Human factors in computing systems:
celebrating interdependence. New York, NY, USA: ACM, pp. 245-251. Available at:
http://doi.acm.org/10.1145/191666.191755.
Krasner, G. & Pope, S., 1988. A Description of the {Model-View-Controller} User Interface Paradigm in the Smalltalk-80 System. Journal of Object Oriented Programming, 1(3), pp.26-49. Available at:
http://citeseer.ist.psu.edu/krasner88description.html.
Landau, V., 2001. “Formative Evaluation Planning”, in Developing an Effective Online Class, 2001.
Available at: 1. http://www.roundworldmedia.com/cvc/module10/notes10.html [Accessed
September 29, 2012].
Leuf, B. & Cunningham, W., 2001. The Wiki Way - Quick Collaboration on the Web 1st Editio., Addison-
Wesley Professional.
Li, N. & Wu, D.D., 2010. Using text mining and sentiment analysis for online forums hotspot detection
and forecast. Decision Support Systems, 48(2), pp.354-368. Available at:
http://linkinghub.elsevier.com/retrieve/pii/S0167923609002097 [Accessed March 2, 2012].
Lindqvist, J. et al., 2011. I‟m the mayor of my house: examining why people use foursquare - a social-
driven location sharing application. In Proceedings of the 2011 annual conference on Human factors in computing systems. New York, NY, USA: ACM, pp. 2409-2418. Available at:
http://doi.acm.org/10.1145/1978942.1979295.
94
Liu, Y., Alexandrova, T. & Nakajima, T., 2011. Gamifying intelligent environments. Proceedings of the
2011 international ACM workshop on Ubiquitous meta user interfaces - Ubi-MUI ’11, p.7.
Available at: http://dl.acm.org/citation.cfm?doid=2072652.2072655.
Maeda, K., Ma, X. & Strassel, S., 2008. Creating Sentence-Aligned Parallel Text Corpora from a Large
Archive of Potential Parallel Text using BITS and Champollion. In LREC. European Language
Resources Association. Available at: http://dblp.uni-
trier.de/db/conf/lrec/lrec2008.html#MaedaMS08.
Marsden, G., Maunder, A. & Parker, M., 2008. People are people, but technology is not technology. Philosophical Transcations of the Royal Society. Available at:
http://pubs.cs.uct.ac.za/archive/00000466/01/fulltext.pdf.
Mathews, B., 2006. Do you facebook? Colledge and Research Libraries News, (May), pp.306-307.
Available at: http://crln.acrl.org/content/67/5/306.full.pdf [Accessed September 4, 2012].
McCluskey, A. & Lalkhen, A.G., 2007. Statistics IV: Interpreting the results of statistical tests.
Continuing Education in Anaesthesia, Critical Care & Pain, 7(6), pp.208-212.
McMillan, W., 2007. Understanding diversity as a framework for improving student throughput.
Education for Health, 20(3), p.71.
Mikkonen, T. & Taivalsaari, A., 2009. Creating a mobile web application platform: the lively kernel
experiences. In Proceedings of the 2009 ACM symposium on Applied Computing. Ney York, USA,
pp. 177-184. Available at: http://dl.acm.org/citation.cfm?id=1529321 [Accessed September 4,
2012].
Muntean, C., 2002. Raising engagement in e-learning through gamification. The 6th International
Conference on Virtual Learning,Cluj-Napoca, Romania, (1). Available at:
http://www.icvl.eu/2011/disc/icvl/documente/pdf/met/ICVL_ModelsAndMethodologies_paper42.pd
f [Accessed May 14, 2012].
Nielsen, J., 1994. Enhancing the explanatory power of usability heuristics. Conference companion on Human factors in computing systems - CHI ’94, p.210. Available at:
http://portal.acm.org/citation.cfm?doid=259963.260333.
Nielsen, J., 2000. Novice vs. Expert Users. Alertbox. Available at:
http://www.useit.com/alertbox/20000206.html [Accessed October 21, 2012].
Nielsen, J., 1993. Usability Engineering, Morgan Kaufmann.
Nielsen, J., 2005. Usability Heuristics. Available at:
http://www.useit.com/papers/heuristic/heuristic_list.html [Accessed September 26, 2012].
Palser, S., 2004. Mobile Bookkeeping Application for Micro Entrepreneurs in the Developing World.
Paul, S. & Hong, L., 2012. Who is Authoritative? Understanding Reputation Mechanisms in Quora. Proceedings of the Collective Intelligence 2012 conference in Cambridge, (2010). Available at:
http://arxiv.org/abs/1204.3724 [Accessed May 14, 2012].
95
Puah, C. & Abu Bakar, A.Z., 2011. Strategies for community based crowdsourcing. 2011 International
Conference on Research and Innovation in Information Systems, pp.1-4. Available at:
http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6125717.
Rainie, L. & Tancer, B., 2007. Wikipedia: When in Doubt, Multitudes Seek It Out.
Rudd, J., Stern, K. & Isensee, S., 1996. Low vs. high-fidelity prototyping debate. interactions, 3(1),
pp.76-85. Available at: http://doi.acm.org/10.1145/223500.223514.
Rukhin, A. et al., 2001. A statistical test suite for random and pseudorandom number generators for
cryptographic applications,
Sartorio, A., 2009. First Approximation to DHD Design and Implementation. Clei electronic journal,
12(1), pp.1-9. Available at: http://www.fceia.unr.edu.ar/~mcristia/publicaciones/clei2009.pdf
[Accessed May 14, 2012].
Scholtz, J., 2004. Usability Evaluation. Available at:
http://www.itl.nist.gov/iad/IADpapers/2004/Usability Evaluation_rev1.pdf [Accessed September 3,
2012].
Seong, D.S.K., 2006. Usability guidelines for designing mobile learning portals. In Proceedings of the
3rd international conference on Mobile technology, applications & systems - Mobility ’06. New York, New York, USA: ACM Press, p. 25. Available at:
http://portal.acm.org/citation.cfm?doid=1292331.1292359.
Seow, S.C., 2005. Information theoretic models of HCI: a comparison of the Hick-Hyman law and Fitts‟
law. Hum.-Comput. Interact., 20(3), pp.315-352. Available at:
http://dx.doi.org/10.1207/s15327051hci2003_3.
Shiraishi, M., Washio, Y. & Takayama, C., 2009. Using individual, social and economic persuasion
techniques to reduce CO 2 emissions in a family setting. Persuasive Technology, Fourth
International Conference, PERSUASIVE 2009, Claremont, California, USA. Available at:
http://dl.acm.org/citation.cfm?id=1541967 [Accessed May 1, 2012].
Shneiderman, B., 1992a. Tree visualization with tree-maps: 2-d space-filling approach. ACM Transactions on Graphics, 11(1), pp.92-99. Available at:
http://portal.acm.org/citation.cfm?doid=102377.115768.
Shneiderman, B., 1992b. Tree visualization with tree-maps: 2-d space-filling approach. ACM
Transactions on Graphics, 11(1), pp.92-99. Available at:
http://portal.acm.org/citation.cfm?doid=102377.115768.
Shneiderman, B. & Plaisant, C., 2005. Designing the User Interface- Strategies for Effective Human-
Computer Interaction Edition, 4., Pearson Addison-Wesley.
Smith, S. L., & Mosier, J.N., 1986. Guidelines for designing user interface software., Bedford,
Massachusetts, USA.
96
StackOverflow, 2012. Stackoverflow- Colloborativelyy edited Q&A for programmers. Available at:
www.stackoverflow.com [Accessed July 30, 2012].
Sulaiman, J. et al., 2009. Implementing Usability Attributes In E-learning System Using Hybrid Heuristics. 2009 International Conference on Information and Multimedia Technology, pp.189-193.
Available at: http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5381220 [Accessed
July 28, 2012].
Suleman, H., 2008. Automatic marking with Sakai. Proceedings of the 2008 annual research conference
of the South African Institute of Computer Scientists and Information Technologists on IT research in developing countries riding the wave of technology - SAICSIT ’08, pp.229-236. Available at:
http://portal.acm.org/citation.cfm?doid=1456659.1456686.
Sweetser, P. & Wyeth, P., 2005. GameFlow : A Model for Evaluating Player Enjoyment in Games. , 3(3),
pp.1-24.
Tazzoli, R., 2004. Towards a semantic wiki wiki web. In In Demo Session at ESWC. Heraklion , Greece. Available at: http://www.tecweb.inf.puc-rio.br/semweb/space/Platypus+Wiki/platypuswiki.pdf
[Accessed July 30, 2012].
Telono, 2012. Telono User-Centered Design Process. Available at:
http://www.telono.com/en/services/usability/ucd-process [Accessed September 4, 2012].
Tullis, T.S., 1984. A Computer-Based Tool for Evaluating Alphanumeric Displays. In INTERACT 84 -
1st IFIP International Conference on Human-Computer Interaction. London, UK.
Turo, D., 1992. Improving the visualization of hierarchies with treemaps: design issues and experimentation. In Visualization ’92, IEEE Conference. Boston,Massachusetts, USA. Available at:
http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=235217.
Ulster, U. of, 2012. University of Ulster Website. Available at: http://www.ulster.ac.uk/ [Accessed May
13, 2012].
Vaughan-Nichols, S.J., 2010. Will HTML 5 Restandardize the Web? Computer, 43(4), pp.13-15.
Available at: http://dx.doi.org/10.1109/MC.2010.119.
Wikipedia, 2012a. Operating Systems Article. Available at:
http://en.wikipedia.org/wiki/Operating_system [Accessed October 22, 2012].
Wikipedia, 2012b. Wikipedia, The Free Encyclopedia. Available at: http://www.wikipedia.org/ [Accessed
July 30, 2012].
Yee, K.-P., 2003. Peephole displays: pen interaction on spatially aware handheld computers. In
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Fort Lauderdale,
FL, USA.: ACM Press. Available at: http://dl.acm.org/citation.cfm?id=642613.
Zhang, P., 2012. The Next Generation Web. Oulu University of Applied Sciences. Available at:
http://theseus17-kk.lib.helsinki.fi/bitstream/handle/10024/42706/Zhang_Pengfei.pdf?sequence=1.
97
Zucker, D.F., 2007. What does AJAX mean for you? interactions, 14(5), pp.10-12. Available at:
http://doi.acm.org/10.1145/1288515.1288523.
98
11 Appendices
11.1 Paper Prototype 1 Horizontal Prototype 1 screens
11.1.1 Screens
CORE WIKI SCREENS
99
GAME SCREENS
11.2 Paper Prototype 2
11.2.1 Screens
100
CORE WIKI SCREENS
101
Commenting System(Not Implimented)
102
GAME SCREENS
103
11.3 Windows Navigation Diagram of the Gamified Wiki
104
Figure 34: Windows Navigation Diagram - Gamified Wiki
11.4 Description of Evaluation Tasks Task 1: Create Page
User Instruction
“May you please create a page?”
Aim
This task examines the user‟s ability to perform a core tasks in the wiki, which is to add new content.
Vula Wiki Description
The user must first login (as for all tasks), and then click the “edit” menu icon from the home screen (or
any article page). The user must then type in the name of the new page (anywhere in the content box)
whilst apply the special Vula syntax/mark-up to create a page. Thereafter, the subject clicks on the save
button and their new page will be displayed as a new link somewhere on the “home page” which they will
know have viewing.
Gamified Wiki Description
Firstly, the user must login (as for all tasks), and have read the introductory tutorial screen (at least on
first login). Thereafter, the actual task requires the user to click the button “+page” (although can be done
from any screen) , read the create page tutorial (at least on the first login), click the button “continue”,
which takes them to the actual “create page” screen. Here, they enter the page title and content and then
press save, at which point the newly created page will display.
Task 2:
User Instruction
“May you please edit a page?”
Aim
This task examines the user‟s ability to perform a core tasks in the wiki, which is to edit content and to
collaboratively build a wiki with other users.
Vula Wiki Description
The user must click the “edit” menu icon from the home screen (or any wiki article). The user must then
type new content anywhere in the content box. Thereafter, the subject clicks on the save button and their
edit will appear somewhere on the page (from which they clicked on the edit button).
105
Gamified Wiki Description
The user must edit a created page or a viewed page. If the former, the user will carry on from the “create
page” use case and then click the “edit” button for the section they want to edit on the new page(which
will be just one section, since page is new). Likewise, in the latter (i.e. view page path), the user would
have went to view wiki, and selected a page to read and choose the appropriate edit button. Thereafter, an
edit screen will appear and they user must just edit the section content, as the section name will be grayed
out (implying read-only). Then, the page will be displayed reflecting the new edited section.
Task 3:
User Instruction
“May you please add a section at end of a page (for gamified)/paragraph with a heading (for wiki)?”
Aim
Again, this task examines the user‟s ability to perform a core tasks in the wiki, which is to edit content
(by adding sections) and to collaboratively build a wiki with other users.
Vula Wiki Description
The user must click the “edit” menu icon from the home screen (or any wiki article). The user must go to
the end, then type a heading (“either by clicking a heading icon or typing special syntax), and add new
content underneath. Thereafter, the subject clicks on the save button and their section will appear
somewhere on the page (from which they clicked on the edit button).
Gamified Wiki Description
The user must click the create section button (either from an article they are viewing or a newly created
page). Then, the user will proceed to add the section data but should not have to explore the place section
option, as the default location for create section, “at the end”, satisfies the task‟s description. Thereafter,
the user clicks save and the new section will be displayed at the end of the page.
Task 4:
User Instruction
“May you please find out when the first operating system was made?”
Aim
106
Again, this task examines the user‟s ability to perform a core tasks in the wiki, which is to navigate the
wiki to find an article of interest to them. Unlike, the former tasks which test the ability to add content this
test the ability to fetch content.
Vula Wiki Description
From the home screen, the user would need to click on the index page, then the “operating systems”
article and then find out the date of the first operating system development. Although, if they got to the
operating systems page, the task was already recorded as complete because the article actually had
multiple invention dates making it hard to distinguish the answer.
Gamified Wiki Description
From the home screen, the user can click the button “view wiki” or alternatively from any screen the user
would need to click the menu icon, “wiki”. From here, they should select “operating systems” article and
then find out the date of the first operating system development. Again, if they got to the operating
systems page, the task was already recorded as complete because the article actually had multiple
invention dates making it hard to distinguish the answer.
Task 5:
User Instruction
“May you please sign up to a tutorial?”
Aim
Again, this task examines the user‟s ability to perform a core tasks in the wiki, which is to navigate the
wiki to find an article of interest to them as well as to edit that content. Therefore, it tests a combination
of fetching from and adding to the wiki.
Vula Wiki Description
From the home screen, the user would need to click on the index page, then the “tutorial sign up” article
and then click the edit button. Thereafter, they would add a time and their name, anywhere in the page so
as to indicate a booking and then they would click the “save” button.
Gamified Wiki Description
From the home screen, the user can click the button “view wiki” or alternatively from any screen the user
would need to click the menu icon, “wiki”. From here, they should select “tutorial sign up” article and
then click the “edit” button (as the whole page is one section, so only one edit button). Thereafter, they
would add a time and their name, anywhere in the page so as to indicate a booking and then they would
click the “save” button.
107
11.5 The Final Evaluation Grid
Table 11: Final Evaluation Grid
User1 User2 User3 User4 User5
Create Page-
Errors
Create Page-Click
Create Page-
Time
Edit Page-Errors
Edit Page-Click
Edit Page-Time
Add section(at end of page)-
Errors
Add section-Click
Add section-
Time
OS Quiz(First OS made)-
Errors
OS Quiz-Click
OS Quiz-Time
Tut Sign Up-Errors
Tut Sign Up-
Click
Tut Sign Up-
Time
11.6 Interview Questions User Anticipations
1. How do you feel about the product?
2. Does the application do what is expected of it?
108
User Experience
3. What are your likes/satisfactions and dislikes/frustrations?
4. Was there any confusing terminology?
5. How did you find the game elements?
6. Overall ease of use from 1 to 5(Likert Scale)?
Recommendations
7. Do you have any future recommendations or changes?
11.7 Image Icons Used Table 12: Image Icons Used
The following is a list of the icons used in the system. The majority of them are from third-parties
however they are all free at least for non-commercial purposes.
Name Icon URL Reference
Newborn
image
http://www.iconarchive.com/show/character-icons-by-martin-berube/Baby-
icon.html
Tree
maps
http://en.wikipedia.org/wiki/File:GradientGroupedTreemap.jpg
Welcome image
http://www.iconarchive.com/show/emoticons-2-icons-by-artdesigner/welcome-icon.html
Create page icon
http://dryicons.com/icon/shine-icon-set/page-add/
Bronze badge
Created by researcher
109
Silver
badge
Created by researcher
Gold
badge
Created by researcher
Editors badge
http://upload.wikimedia.org/wikipedia/commons/b/b3/Orange_Icon_Edit.svg
Contributors badge
http://www.iconarchive.com/show/keriyo-emoticons-icons-by-deleket/Smiley-zzz-icon.html
Admirer
badge
http://icons.iconarchive.com/icons/deleket/keriyo-emoticons/256/Smiley-
icon.png
Popular
badge
http://www.iconarchive.com/show/keriyo-emoticons-icons-by-
deleket/Smiley-pleased-icon.html
Authors
badge
http://findicons.com/icon/168686/graphic_design
Readers
Badge
http://icons.iconarchive.com/icons/deleket/keriyo-emoticons/256/Smiley-
cool-icon.png
Sample
Medal
http://www.geograph.org.uk/photo/1944243
Points
guage
http://archive.plugins.jQuery.com/project/speedometer
No
Message
http://www.veryicon.com/icon/png/Business/Pretty%20Office/New%20messa
ge.png
Message
Received
http://www.veryicon.com/icon/png/Business/Pretty%20Office/Message%20al
ready%20read.png
Like
button
http://www.clker.com/clipart-thums-up.html
110
11.8 Statistical Data Wilcoxin signed-rank tests are computed on each pairs of data.
Table 13: Average Clicks Stats
TASK1 TASK2
Wilcoxon's W: 0
n: 2
P-value: P > 0.2
Systems
Person Wiki Gami
3 2 3
5 2 3
7 2 2
Wilcoxon's W: 1.5
n: 2
P-value: P > 0.2
Systems
Person Wiki Gami
2 4 1
3 2 2
4 2 2
5 2 2
6 2 2
7 2 2
8 2 5
9 2 2
10 2 2
TASK3 TASK4
Wilcoxon's W: 0
n: 0
P-value: P > 0.2
Systems
Person Wiki Gami
5 2 2
6 2 2
7 2 2
Wilcoxon's W: 3
n: 3
P-value: P > 0.2
Systems
Person Wiki Gami
1 1 2
2 2 3
3 4 2
4 3 3
6 3 3
111
TASK5
Wilcoxon's W: 3
n: 3
P-value: P > 0.2
Systems
Person Wiki Gami
1 1 2
2 2 3
3 4 2
4 3 3
6 3 3
Table 14: Average Task Completion Time
TASK1 TASK2
Wilcoxon's W: 0
n: 2
P-value: P > 0.2
Systems
Person Wiki Gami
1 180 90
3 180 90
5 120 120
7 120 120
Wilcoxon's W: 2
n: 6
P-value:
0.05 < P < 0.10
Systems
Person Wiki Gami
2 180 60
3 180 120
4 120 120
5 60 60
6 120 90
7 120 60
8 120 120
9 72 90
10 60 45
TASK3 TASK4
Wilcoxon's W: 0
n: 2
P-value: P > 0.2
Wilcoxon's W: 1.5
n: 4
P-value: P > 0.2
112
Systems
Person Wiki Gami
5 120 90
6 120 90
7 120 120
Systems
Person Wiki Gami
1 120 60
2 120 90
3 240 60
4 150 180
6 120 120
7 120 120
TASK5
Wilcoxon's W: 0
n: 2
P-value: P > 0.2
Systems
Person Wiki Gami
3 180 90
7 120 60
Table 15: Average Errors per Task
TASK 1 TASK 2
Wilcoxon's W: 0
n: 4
P-value:
P < 0.001
Systems
Person Wiki Gami
1 1 1
2 0 0
3 1 0
4 1 0
5 0 0
6 1 0
7 1 1
8 0 0
9 0 0
Wilcoxon's W: 2.5
n: 4
P-value: P > 0.2
Systems
Person Wiki Gami
1 1 0
2 1 0
3 0 0
4 0 0
5 0 0
6 0 0
7 1 0
8 0 1
9 0 0
113
10 1 0
10 0 0
TASK 3 TASK 4
Wilcoxon's W: 6
n: 5
P-value: P > 0.2
Systems
Person Wiki Gami
1 1 1
2 0 0
3 0 1
4 1 0
5 0 0
6 1 0
7 0 1
8 1 1
9 0 0
10 1 0
Wilcoxon's W: 8
n: 7
P-value: P > 0.2
Systems
Person Wiki Gami
1 0 0
2 0 0
3 1 0
4 0 1
5 1 0
6 0 0
7 0 1
8 1 0
9 1 0
10 1 0
TASK 5
Wilcoxon's W: 2
n: 3
P-value: P > 0.2
Systems
Person Wiki Gami
1 1 1
2 1 1
3 0 0
4 0 1
5 1 0
6 1 0
7 1 1
8 1 1
9 1 1
10 1 1