selection of theses

A synopsis of Educational Technology R&D:

Lessons learned from a selection of theses

submitted to Open University of Cyprus

Dr. Panagiotis Zaharias

Open University of Catalonia - 4/6/2015

Experience from master theses supervision at

Open University of Cyprus

Mentored/supervised more than 20 students in their

master thesis along topics such as:

- Usability and UX methods,

- Serious games / Game-based learning,

- Online and video games,

- Virtual worlds,

- Gamification,

- Social media in education etc.

Assessing long-term UX in the context of

MMOGs

Time spans of User Experience (UX)

Source: http://www.allaboutux.org/files/UX-WhitePaper.pdf

UX in the long term: an important but

neglected issue

While the importance of temporality has been

repeatedly highlighted in user experience research, it has rarely been systematically addressed.

…Due to the effort involved in conducting longitudinal

studies.

or a lack of sufficient interest, induced by a belief that

motivating prolonged use does not necessarily lead to

increased commercial revenues.

Context of the study Based on:

Thesis submitted by Moschou, Eirini - thesis title: “Development of a UX

evaluation method for Virtual Learning Environments” - January 2014

Why long term UX in MMOGs?

Shortage of relevant studies so far

Players usually have a long lasting relationship with the game they play

The main objective of the study:

To investigate, understand and interpret the changes of players’ experience over time

The method: A new version of UX Curve

UX Curve: A retrospective method for assessing long term UX

proposed by Kujala et al. (2011).

Based on drawings made by users

Cost efficient as a long term usage can be covered in a single session

Results give an overview of the most relevant experiences

Memories are as important or more than actual experiences

UX Curve templates

Context of empirical study

The game: League of Legends

The players: 9 students aged between 18-20 years old who had been

playing LoL for periods ranging from 6 to 16 months (mean: 12.5

months SD: 3.1 months)

The new version of UX Curve:

UX Curve

• General UX

• Attractiveness

• Ease of use

• Utility

• Usage Volume

A new UX Curve (MMOG version)

• General UX

• Degree of usage

• Ease of use

• Immersion

• Social interaction

• Engagement

Indicative results (1)

A total of 54 curves were collected. Most of the curves were

improved revealing thus an increasing user experience

General UX Curve for each user ID The immersion UX Curve for each user ID


In order to draw the curves, players described the factors that

improved their experience over time or caused it to deteriorate

Reason Categories Positive Negative

Usability 6 4

Utility 3 2

Aesthetics 7 0

Gameplay 5 3

Challenge 6 0

Social interaction 7 5

Interest 6 0

Miscellaneous 3 0

Sum 43 14

Table2 -The categories of the reasons for general UX curve

Reason Categories Positive Negative

General UX 43 14

Ease of use 41 26

Immersion 43 10

Social interaction 38 12

Engagement 32 10

Total 197 72

Table 1- Number of reasons for general and specific UX dimensions


All except for two “Ease of use” curves were improving or stable, with

pragmatic-related reasons to be the most frequent.

most negative perceptions related to usability reasons

Overall, most of the issues influencing the long-term user experience

in the game were related to non-pragmatic issues such as fun,

immersion, challenge, interest and control.

Key Takeaways

It is crucial to measure UX in the long term

Retrospective methods can greatly help in understanding how UX

changes over time

Memories are as or more important than actual experiences

UX Curve is a cost efficient and effective method to use for

analyzing and understanding long term UX

The new proposed version of UX Curve (customized to gaming

environments) seems to be a valid and effective method for

assessing long term UX of MMOG players

It can be applied to many other contexts for evaluating

products/services/systems

Development of an E-learning UX

Measurement System

Context of the study

Based on:

Thesis submitted by Evangelos Loutas - thesis title: “Development

of an E-learning UX Measurement System” - May 2013

Usability and UX measurement in e-learning applications

Studies show that usability/UX is a crucial success factor in e-

learning and facilitates learners to achieve learning objectives and

gain knowledge effectively and efficiently


To develop a web-based system that will measure UX, according to

Mo2L usability evaluation method.

Mo2L questionnaire

The Mo2L instrument can be used to measure different dimensions

of the e-learning user experience,

i.e. from typical usability attributes such as content, navigation and

learnability etc. to affective learning issues such as motivation to

learn.

A validated questionnaire with 49 items along 8 scales:

Content, Learning design and support, Visual design, Navigation,

Accessibility, Interactivity, Self-assessment & Learnability and

Motivation to Learn

UX e-learning measurement

Towards quantification of e-learning usability perceptions

The method provides calculation of a total “e-learning usability

value” which takes into consideration all items on the questionnaire

and reflects a global usability score of the e-learning

application.

In addition calculations are made along all the different scales of the

questionnaire presenting thus quantifiable information for all the

aspects of an e-learning application.

Main workflow of the system as developed

1. Interested researcher or practitioner has to register

2. A new instance of the system is created and a link to the researcher is sent. Researcher can use this link to initiate his/her usability study by calling users/participants.

3. The enquiry participants will be provided with a 49-item questionnaire that contains also some demographics questions.

4. All questions are mandatory. Failure to reply to a question will result to the invalidity of the participant's response.

5. After a certain period of time and a fair number of enquiry participants the questionnaire link will close and further information will be given to registered users on how to access the reports containing the usability measures.

6. The reports are extracted by using an automated procedure. Custom report generation is possible upon request to the administrator.

The main features of the system

Reports for the mean Global E-learning Usability Score (GEUS)

Reports for all the respective Mo2L scales: mean Motivation to Learn

score, mean navigation score, etc.

Reports for individual scores and

Reports for open questions

Impact and future research

From May 2013 – today more than 70 researchers and practitioners

around the world have expressed their interest to use the system in

their usability and UX studies.

Quite recently five instances of the system were open and being

used by researchers from US, England and Portugal.

Shortly we are planning a redesign of the system in order to make it

more robust and usable

Check out the site:

http://www.elearning-usability.com/index2.html





Development of a serious 3D game as a

tool for organizational learning

Context of the study

Based on:

Thesis submitted by Demetrios Mouzouros, thesis title: “Development

of a serious 3D game as a tool for organizational learning” - August

2012

Organizational learning processes and human development

New methods, techniques and tools are needed to effectively support

process such as onboarding, etc.


To develop a serious game and use it as an organizational learning

tool

To empirically test it for onboarding employees in a real company environment

The development of the “Knowledge Donor”

The game takes place in a 3D world that simulates a business

environment and gives the user a realistic feeling of an actual

corporate training.

The game scenario is based on two main pillars:

a) the existing procedure that takes place in an insurance company

for the orientation of new employees, and

b) the fact that a percentage of the profit of this company goes to

charity (strong relations of the company with a scholarship

foundation)

The main game scenario

The player is a new employee. Other employees of the company will

take the player through an orientation and initial training session

Content about the company creation and history, the products and

marketing routines and the personnel of the company.

The game consists of three stages.

At the end of each stage, the player is tested for the “knowledge”

that has acquired within specific time limits

Successful completion of the tests will reward the player with money,

all of which is collected as a donation to the scholarship foundation.

Empirical study

The “Knowledge Donor” game was evaluated within a period of 20

days where the employees of the insurance company had the

chance to play the game.

After the play sessions, interviews were conducted (by the student)

to get feedback from the players.

Along with the interviews, the users were asked to fill-in a user-

experience questionnaire which was based on the AttrakDiff Lite

Questionnaire.

The questionnaire measures hedonic qualities and pragmatic

qualities through 10 items that are presented in a 7-point semantic

differential scale

Some key findings

Reactions and comments of the users revealed a positive approach towards

this new intervention and a great interest in playing the game.

Almost 80% said that this could be a new and effective method to support

onboarding processes in the company.

The same users mentioned that the game helped them refresh their

memory regarding certain things about business processes.

However only 50% would choose this method over the traditional face-to-

face training sessions.

Users also reported some problems with their interaction with the 3D world.

They faced mainly navigation problems and they complained regarding

the content

Main limitations and future prospect

Sample of participants

Duration of the study

Future implementations should be based on detailed UX driven

process of capturing requirements

Future implementation must involve the real target users: new

employees that join the company

LEARNING GEOGRAPHY THROUGH SERIOUS GAMES:

THE EFFECTS OF 2-D AND 3-D GAMES ON LEARNING

EFFECTIVENESS, MOTIVATION TO LEARN AND USER

EXPERIENCE

Context of the study Based on:

Thesis submitted by Ioanna Chatzeparaskeuaidou, thesis title:

“Development of an educational game as a tool for learning geography in

elementary schools” - August 2012

Design of 2-dimensional and 3-dimensional serious games in education

Limited knowledge and relevant empirical research in formal educational

settings that compares their effectiveness


To develop an educational game for learning geography in elementary

schools – develop two versions a 2D and 3D

To empirically compare their effects on learning effectiveness, motivation to learn, and UX.

Empirical study

Research question: What effects do 2D and 3Dversions of the game have

on learning effectiveness, motivation to learn and user experience?

The game:

Design of two versions of the game with Scratch platform and Kodu.

The game scenario: quite simple and structured around an interactive

map of Greece.

The game follows a question and answer method: it prompts the player to

choose one geographical region and continues with several related

questions that are region specific. This is happening repeatedly for other

regions as well and the total number is twenty questions.

Snapshots of the game

Experimental process

Participants were students from eight elementary schools located in areas of

Northern Greece.

159 students participated in the experiments, 94 girls (59.1%) and 65

boys (40.9%), attending the 5th and 6th grade, from 16 urban and

suburban parts.

The age ranged between 10 and 12 years old.

Data were collected through questionnaires and mini interviews:

a demographic survey was employed along with a multiple choice test for

assessing learning effectiveness on Geography and two scales

measuring motivation to learn and user experience

Experimental process

Set up: A control group with students playing the 2D version of the game

and the experimental group with students playing the 3D version (Students

were randomly selected based on the "lottery" method)

A mixed between-within subjects ANOVA model:

A session (2: Pretest, Posttest) x design modality (2: 2D, 3D) ANOVA was

conducted with session as the within-subjects factor and design modality

as the between subjects factor

The experiments took place at the computer labs in 8 elementary schools.

Participation of the students and their teachers was on voluntary basis

during the whole process

Experimental process and results

Before running the mixed ANOVA an independent samples t-test for

differences between the mean scores of male and female subjects was

performed. - No statistical significant differences were found.

Learning effectiveness:

It was found that there is a significant interaction between design modality and

session, Wilks Lambda = .84, F (1, 149) = 27.08, p < .0005, partial eta squared = .15.

There was a substantial main effect for session (pretest and posttest), Wilks Lambda =

.72, F (1, 149) = 57.05, p < .0005, partial eta squared = .27, with both groups showing

an increase in learning effectiveness scores across the two time periods.

Results

Learning effectiveness: players in 2D group performed better in geography

knowledge tests than players in 3D group.

Authors observed that:

students in 2D group were more focused on the successful completion of

the game while

students playing the 3D game were almost distracted from the visual

design of the game trying to learn to navigate and willing to explore the

elements of the environment beyond the learning focus of the game

Results

Motivation to learn:

Results revealed that there is no significant interaction between design

modality and session, Wilks Lambda = .97, F (1, 149) = 3.19, p = .076,

partial eta squared = .021.

There was a substantial main effect for session (pretest and posttest), Wilks

Lambda = .81, F (1, 149) = 33.69, p < .0005, partial eta squared = .184, with

both groups showing a decrease in motivation to learn scores across the

two time periods

Results


We observed that students entered the experimental process with a great

enthusiasm

while they were engaged with game activities they oriented themselves

more towards fun than learning.

This may explain the fact that motivation to learn decreased after the end

of the experiment. Confirmed in other studies as well (Yang 2012, Annetta

et al. 2009), as the novelty effect fades motivation gets lower.

Results


The main effect comparing the two versions of game was also significant,

F (1, 149) =7.71, p = .006, partial eta squared = .049, suggesting a

difference between the two design modalities.

We see that players in 3D group showed greater motivation to learn than

those who played the 2D version.

Results

UX:

There is no significant interaction between design modality and session,

Wilks Lambda = .99, F (1, 149) = .28, p = .595, partial eta squared = .002.

There was a substantial main effect for session (pretest and posttest),

Wilks Lambda = .77, F (1, 149) = 42.46, p < .0005, partial eta squared =

.222, with both groups showing a decrease in user experience scores

across the two sessions

Observation: Students had great expectations in terms of anticipated UX in

both groups

Results

UX:

The main effect comparing the two versions of game was also significant,

F (1, 149) =4.75, p = .031, partial eta squared = .031, suggesting a

difference between the two design modalities.

players in 3D group expressed a higher degree of user experience than

the players in the 2D group.

We argue that 3D learning environments are supposed to enhance the user

experience in terms of flow, presence, etc.

We attribute this effect on the perceived hedonic qualities of 3D

environments such as novelty and greater fidelity.

Future research

Longer experiments are needed

Different types of knowledge tests

Design modalities (2d and 3d) can be combined in a single game

The games could be designed in a multiplayer mode with a different

pedagogical focus where activities will require collaboration for successful

completion of the game.

Some papers associated with the theses

[C21] Chatzeparaskeuaidou, I. and Zaharias, P. Hedonic and pragmatic qualities as predictors for motivation to learn in serious educational games. Accepted at the Foundations of Digital Games 2013, Workshop on Games for Learning.

[J17] Chatzeparaskeuaidou, I. and Zaharias, P. The effects of a 2D and 3D game on learning effectiveness and motivation. Submitted to Journal of Educational Technology & Society

[C24] Moshou, E. and Zaharias, P. (2013). The UX Curve revisited: Assessing long term UX for games. Accepted at the Workshop on Designing Gamification: Creating Gameful and Playful Experiences at CHI 2013 - Conference on Human Factors in Computing Systems.

[J19] Zaharias, P. and Moschou, E. Drawing Curves for assessing long term UX of Massive Multiplayer Online Games. Submitted to Interacting with Computers

[J16] Mouzouros, D. and Zaharias, P. Development of a 3D serious game for knowledge management: an empirical investigation. Submitted to Journal of Virtual Worlds Research

Thank you very much !

Any Questions?

Contact info:

[email protected] | [email protected]

http://about.me/panagiotis_zaharias

http://gr.linkedin.com/pub/panagiotis-zaharias/0/bb6/a91

mailto:[email protected]

mailto:[email protected]

http://about.me/panagiotis_zaharias




selection of theses

Documents