ANALYSIS OF IMMERSIVE VIRTUAL REALITY VS. DESKTOP 3D GAMES

Thesis Presented

by

Prasad Raut

to

The College of Arts, Media and Design

In partial fulfillment of the requirement for the Degree of Master of Science in Game Science

and Design

Northeastern University

Boston, Massachusetts

December, 2018

2

ANALYSIS OF IMMERSIVE VIRTUAL REALITY VS. DESKTOP 3D GAMES

by

Prasad Raut

ABSTRACT

Virtual Reality(VR) has become popular in the past few years. Due to this, besides video games,

VR is now being used in applications designed in fields of education, fitness, healthcare, etc. to

improve the effectiveness of those applications. In this research, a comparative study between

immersive virtual reality and desktop real-time 3D was made to determine the various attributes

and which medium of game was more effective. An analysis of quantitative and qualitative data,

gathered by means of mixed methods, was performed on the responses of participants playing

these immersive and non-immersive versions of the game and the results are discussed in the paper.

Keywords: Virtual Reality, Desktop 3D, Think-Aloud Protocol, Video Analysis, Sentiment

Analysis

Submitted in partial fulfillment of the requirements

for the degree of Master of Science in Game Science and Design

in the Graduate School of the College of Arts, Media and Design of

Northeastern University

December, 2018

3

ACKNOWLEDGMENTS

I would first like to thank my thesis advisor Dr. Celia Pearce for helping me with the qualitative

analysis of the study and giving me feedback that helped me extensively with this research. I

would next like to thank my thesis instructor Christoffer Holmgård Pedersen for the insightful

feedback, words of encouragement, and being available when I needed your help. I would also

like to thank Jason Duhaime for helping me use the Usability Lab at Northeastern University and

providing me support in setting up all equipment and software for conducting this study and

gathering the data.

I would like to thank all my participants without whom I wouldn't have been able to collect

valuable data for this research study. I also want to thank Jennifer Gradecki, who was diligent in

helping me recruit enough participants for this thesis. I would like to thank my classmates and

friends in Game Science and Design department who gave me valuable suggestions and advice

on my research topic. Their kindness and enthusiasm in supporting my research and their selfless

helping provided me with great encouragement on the study. Lastly, I want to thank my parents,

Sheela and Sunil Raut, for always supporting me.

4

TABLE OF CONTENTS

Abstract 2

Acknowledgments 3

1. Introduction 5

2. Background

2.1 Virtual Reality and Fitness 6

2.2 Treatment of Acrophobia 7

2.3 Desktop Virtual Reality and Learning Outcomes 8

3. Methodology

3.1 The Game 9

3.2 Participants 10

3.3 Think-aloud Protocol and Video Analysis 10

3.4 Survey 12

4. Results

4.1 Analysis of Survey Data 13

4.2 Analysis of Video Data 18

5. Discussion

5.1 Quantitative Data 22

5.2 Qualitative Data 23

6. Conclusion 25

7. References 26

8. Appendix A 28

5

1. INTRODUCTION

Virtual Reality (VR) is a technology that has become extremely popular in recent years. Due to

the availability of cheap VR headsets which can be bought around 20 dollars, the number of VR

games is on the rise (Telegraph Reporters, 2018). To take advantage of this trend, educators are

seeking different ways to create attractive applications in the form of interactive games that will

motivate and engage students or users in learning different topics (Virvou and Katsionis, 2008).

Furthermore, the gaming environments can encourage a constructionist approach to learning.

According to Papert (1980), such constructionist approaches motivate children to acquire

knowledge through creative and interactive experiences.

The objective of this research is to analyze a virtual reality game designed for educational purpose

to determine and understand the factors that make them favorable to players. For this study, an

educational game was chosen, and a playtest was conducted on two groups of participants to

determine these factors. These participants responses were collected by means of mixed methods.

The primary goal of this research is to evaluate these findings to reveal the special characteristics

that may hold the key to their success among players.

6

2. BACKGROUND

2.1 Virtual Reality and Fitness

Virtual Reality (VR) applications can be divided into two categories: immersive VR and Desktop

real-time 3D. In Immersive VR, users wear head-mounted displays and are often completely

surrounded by enclosed virtual environment, whereas in Desktop 3D user’s experiences are

limited to what they see on their desktop or laptop display monitors and what they hear from

their speakers (Mills and Noyes, 1999). Throughout this research, we will be focusing on both

the Desktop 3D version of the application as well as the immersive VR version using the head-

mounted display.

The immersiveness of VR applications can be exploited for fitness. Tuveri et. al. (2016) made

use of this immersion along with gamification techniques to improve fitness. For their research,

they developed an immersive virtual environment and incorporated gamification features in it

using Unity 3D game engine. They also created a hardware setup using consumer-level devices

such as a regular exercise bike, Oculus Rift VR headset and a Microsoft Kinect device to track

users’ movements and replicate them in the virtual environment. They performed a research

study for evaluating two different aspects of their game prototype. The first goal was to

determine whether the users enjoyed more physical activity with gamification elements while

interacting with immersive VR environments. The second goal was to provide a qualitative

assessment of the different gamification techniques used by them, according to three dimensions:

usefulness, fun and motivation. The results of a user test showed that such gamification elements

7

along with the immersive virtual environment, providing the ability to change viewpoint by

moving their heads, increased the user's enjoyment during the physical activity.

2.2 Treatment of Acrophobia in Virtual Reality

In their research study of Treatment of acrophobia in virtual reality: The role of immersion and

presence, Krijin et. al. (2004) made use of Virtual Reality Exposure Therapy (VRET) for

treatment of patients with acrophobia. The immersion of VRET was adjusted by using two

different versions, a Head Mounted Display (HMD) for low immersion and a Computer

Automatic Virtual Environment (CAVE) for high immersion. The results of this study showed

that the VRET was more helpful than no treatment, whereas no significant differences in the

effectiveness of VRET between HMD and CAVE were observed in the study. The VRET

systems in both versions proved effective in treatment on anxiety, (behavioral) avoidance and

attitudes towards heights. From his study, it can be concluded that immersive virtual reality can

be used in software applications which are created for the purpose of treating negative emotional

functions.

8

2.3 Desktop Virtual Reality and Learning Outcomes

The study conducted by Lee et. al. (2010) examined how desktop real-time 3D not only

influences but enhances learning. The results of the study provide a guideline for VR software

developers to improve and strengthen the learning effectiveness of Desktop 3D applications

implemented for learning purposes. Different relevant constructs like usability, presence,

motivation, cognitive benefits, control and active learning, reflective thinking, learning

outcomes, and student characteristics were analyzed to investigate how Desktop 3D enhances

learning.

The results backed the indirect effect of Desktop 3D features to the learning outcomes, which

was mediated by the interaction experience and the learning experience. Learning experience

which was individually measured by the psychological factors like presence, motivation,

cognitive benefits, control and active learning, and reflective thinking strongly affected the

learning outcomes in the Desktop 3D-based learning environment. Desktop 3D-based learning

could provide students with different learning styles and spatial abilities. This research

contributed an initial theoretical model of the determinants of learning effectiveness in a desktop

3D-based learning environment.

9

3. METHODOLOGY

To compare the effectiveness of immersive VR, Titans of Space 2.0 was selected for the research

study. It is a virtual reality educational game based on the Solar System. For conducting this

research study, the game was played by two different groups of participants, one group played

the game using Oculus Rift VR headsets and controllers and the other group played the same

game using regular desktop computers with keyboard and mouse. The use of various mixed

methods mentioned below was made to collect data for analysis.

3.1 The Game

Titans of Space 2.0 by DrashVR LLC is an immersive VR space education app created for HTC

Vive and Oculus Rift VR headsets. Recently, it has also been launched for Google Cardboard

VR using Android and iOS. Titans of Space 2.0 (Figure 3.1) is a VR game that allows people to

journey through our solar system. This game is useful for educational purposes since you can use

it in conjunction with teaching students about the composition of each planet, the revolutions

around the sun, how many moons each has as well as gravity among so much other important

and useful information (DrashVR LLC, 2016). Such teaching methods become more interesting

when the students can view and interact while learning about the solar system. What started out

as a game can help students reach a higher level of engagement than traditional learning.

10

Figure 3.1 Titan of Space 2.0 gameplay.

3.2 Participants:

A total of 26 participants over the age of 18 years volunteered for this study. During the study

participants were alternatively selected at random to play either the VR version of the game or

the desktop version for a duration of 15 to 30 minutes. Hence, 13 participants played the game in

VR and the other 13 played on desktop computer. The names of the participants were kept

anonymous throughout this research study and their responses were identified using a unique

identity code assigned to them before conducting the research study.

3.3 Think-Aloud Protocol and Video Analysis:

For video analysis, the use of Morae software in the usability lab at Northeastern University was

made. The participants were instructed to think-aloud while they played the game and the video

transcripts of them playing the game as well as their on-screen activity, speech and mouse

11

movement were recorded with the Morae Recorder. To note their experience qualitative analysis

of these video transcripts were performed with the help of the Morae Observer and Manager.

Figure 3.3.a Screenshot of Morae observer for Oculus version of game

Figure 3.3.b Screenshot of Morae Observer for desktop version of the game

12

The Morae Observer facilitates researchers to watch the participant's experience, take notes, and

flag tasks in real-time, and the Morae Manager is used to view and analyze Morae recordings,

automatically perform calculations, visualize data, and create highlight videos to share with

stakeholders (TechSmith, 2018). Figures 3.3.a and 3.3.b show screenshots of the game in Morae

Observer. The Morae Manager was later used to save and export the video transcripts to Google

Drive cloud storage. Analysis and visualization of the data from the video transcripts was

performed later.

3.4 Survey:

After the playtest, the participants from both groups filled out the same survey which contained a

5-point and 7-point Likert scale questionnaires regarding their thoughts about the game they

played. These survey questions were created and analyzed using Qualtrics and Tableau software.

Qualtrics is an easy to use web-based survey tool for conducting survey research, evaluations

and other data collection activities whereas Tableau is a data visualization tool created by

Tableau Software which is a used to create interactive data analysis and visualization.

An open-ended question at the end of the survey was given asking the participants to briefly

describe their experience playing the game. Sentiment Analysis of these descriptions given by

each group was conducted in R using the Tidytext package. All these software tools were used to

analyze data and create charts and tables to visualize the data.

13

4. RESULTS

4.1 Analysis of the survey data:

The results for the 5-point and 7-point Likert scales were noted and the median of their score was

calculated. The median value described various aspects of the game the participants felt from

both groups like graphics, controls, learning, immersion, etc.

Figure 4.1.a Comparison of median values of participants for Desktop and Oculus version

The bar graph above (Figure 4.1.a) shows comparison between the median values of points for

the 5-point Likert scale. It can be observed that participants from both groups enjoyed playing

the game, found the graphics to be appealing and the controls were easy for them to understand

and use. The participants who played the Oculus version of the game felt less consciously aware

14

of their surroundings than those who played the Desktop version. Also, the participants from the

desktop group felt that time slowed down, indicating that the game was too long to play while

those from Oculus group felt that the game was relatively of short duration.

Figure 4.1.b Comparison of median values of participants for Desktop and Oculus version

The graph above (Figure 4.1.b) shows comparison between the median values of points for the 7-

point Likert scale. Participants from both groups felt that they learned new facts and information

in the game. Participants from the Oculus group comparatively to the Desktop group were more

focused on the game and felt that they were experiencing the events in the game rather than just

doing tasks. Also, Oculus group participants felt that the game environment was more interactive

and more immersive than the Desktop group participants.

15

For the sentiment analysis, the bing lexicon from the Tidytext package was used. The bing

lexicon was created by Bing Liu and collaborators and is used to identify and categorize words in

a binary fashion into positive and negative categories (Silge and Robinson, 2018). The lexicons

in the Tidytext package were created by either crowdsourcing (using, for e.g. Amazon

Mechanical Turk) or by the efforts of the author and collaborators. These lexicons were verified

and validated using some combination of crowdsourcing, restaurant or movie reviews, or Twitter

data.

Figure 4.1.c Text Analysis flowchart using Tidytext for Sentiment Analysis

As described in the flowchart (Figure 4.1.c) above, the raw text data from the description

sentences of experiences from each group was unnested into tokens of single words which were

then placed into a table. This table of unnest token words was then compared with the bing

lexicon using an inner join and the matching words were found for positive and negative

sentiments. The group_by transformation from the Dplyr package was used on this joined table

to aggregate the matching cases to be a summarized count of the positive and negative words.

16

Later this summarized table was visualized using the ggplot package in R to derive meaningful

insights as shown in figures 4.1.d and 4.1.e below.

Figure 4.1.d Positive and Negative words for experience in Desktop version

The bar chart above (Figure 4.1.d) shows the most number of positive and negative words

occurring in the description of experience from players who played the game on desktop

computer. The negative words hard, distracting and bored occurred most number of times

indicating that some of the participants were having issues with controls, found the in-game

music to be distracting while reading facts, and were bored while playing the game on a desktop

computer. However, the positive word enjoy was observed the most implying that most of the

participants really liked and enjoyed playing the game on desktop computer.

17

Figure 4.1.e Positive and Negative words for experience in Oculus version

The bar chart above (Figure 4.1.e) shows the most number of positive and negative words

occurring in the description of experience from players who played the game on the Oculus VR

headset. The positive words like incredible, impressive, beautiful, amazing implies that the

participants were mostly likely drawn to the aesthetics of the game in VR. The negative words

scary and sickness occurred the most indicating participants also found the stars appearing in

front of them to be eerie and intimidating to look at, while some participants were not

comfortable with the movement in VR and felt motion sickness.

18

4.2 Analysis of video data:

Figure 4.2.a Comparison of median time spent by participants on objects in Desktop and

Oculus version

The time spent by participants on each object in the game was noted for each group (Figure

4.2.a). Since the total time spent by participants is not fixed and varies between 10 to 40 minutes

in the game for both versions of the game, the total time spent is normalized by min-max

normalization using the formula below:

𝑵𝒐𝒓𝒎𝒂𝒍𝒊𝒛𝒆𝒅 𝒕𝒊𝒎𝒆 𝒔𝒑𝒆𝒏𝒕 =𝑻𝒊𝒎𝒆 𝒔𝒑𝒆𝒏𝒕 𝒑𝒆𝒓 𝒐𝒃𝒋𝒆𝒄𝒕

𝑴𝒂𝒙(𝑻𝒊𝒎𝒆 𝒔𝒑𝒆𝒏𝒕) − 𝑴𝒊𝒏(𝑻𝒊𝒎𝒆 𝒔𝒑𝒆𝒏𝒕)

In the above formula, the minimum time spent will always be zero and the maximum time spent

will depend on the participant’s individual total time spent in the game. So, the above formula

can be rewritten as:

19

𝑵𝒐𝒓𝒎𝒂𝒍𝒊𝒛𝒆𝒅 𝒕𝒊𝒎𝒆 𝒔𝒑𝒆𝒏𝒕 𝒑𝒆𝒓 𝒐𝒃𝒋𝒆𝒄𝒕 =𝑻𝒊𝒎𝒆 𝒔𝒑𝒆𝒏𝒕 𝒑𝒆𝒓 𝒐𝒃𝒋𝒆𝒄𝒕

𝑻𝒐𝒕𝒂𝒍 𝒕𝒊𝒎𝒆 𝒔𝒑𝒆𝒏𝒕 𝒃𝒚 𝒑𝒂𝒓𝒕𝒊𝒄𝒊𝒑𝒂𝒏𝒕

After normalizing the time spent per object by each participant, the median time spent per object

was calculated for each group of participants (Table 4.2.a). Since the distribution is not normal,

median of time spent was calculated.

Object Median normalized time

spent in Desktop version

Median normalized time

spent in Oculus version

Earth 0.030 0.045

Mercury 0.020 0.015

Venus 0.010 0.025

Mars 0.025 0.025

Jupiter 0.035 0.040

Saturn 0.020 0.030

Uranus 0.015 0.025

Neptune 0.010 0.020

Pluto 0.030 0.030

Sun 0.030 0.030

Pollux 0.020 0.010

Rigel 0.020 0.020

VY Canis

Majoris

0.020 0.020

Table 4.2.a Median of normalized time spent per object for Desktop and Oculus versions

20

It can be observed that the median time spent for most of the objects in Oculus version of the

game is equal to or higher than that of the desktop version. Except for Mercury and Pollux,

participants from the Oculus group spent equal or more time compared to the Desktop group.

To further investigate the differences in these two groups an independent 2-group Mann-Whitney

U Test was performed in R. Since the total time spent by different participants in each group is

not constant and varies between 10 to 40 minutes, the median of the normalized total time spent

per object for both groups was used as numeric vector inputs in the Mann-Whitney Test function.

The significance value derived for the differences between the two groups (p-value = 0.4799 and

W = 133.5) does not reject the null hypothesis.

Categories Definition

Affect Affect is a concept used in psychology to

describe the experience of feeling or emotion.

Cognition The mental process of acquiring knowledge

and understanding through thought,

experience, and the senses.

Evaluation Player critique and feedback received during

the gameplay session.

Experience Player comments about the in-game events

happening around player.

Game Design Player comments on mechanics, movements,

and controls in the game.

Visual Design Player comments on the graphics and

aesthetics of planets, stars and other objects in

the game.

Table 4.2.b Categories of video coding and their definitions

21

The comments from the video transcript were categorized based on the definitions provided in

table 4.2.b above. The definition for the codes or categories were given according to the

subjective knowledge of the researcher when conducting qualitative analysis of the video

transcript.

Figure 4.2.b Comparison between video codes for Desktop and Oculus versions

From the bar chart (Figure 4.2.d) the cognition category indicated 155 comments from Desktop

version and 130 comments from Oculus version related to Learning from the game. There were

16 comments from the Desktop version and 24 comments from the Oculus associated with

Affect i.e. experiencing emotions and feelings while playing the game. 118 comments from the

22

video transcript of Desktop group and 51 from Oculus group were about Evaluation. Also, 42

comments from the Oculus group were on the experience as compared to 21 comments from

Desktop group. There were 73 comments from the Desktop group and 21 from Oculus group

regarding Game Design, whereas only 19 comments from the Desktop group and 67 from

Oculus group regarding Visual Design.

5. DISCUSSION

5.1 Quantitative data

The quantitative data from the survey indicated that the game in both Desktop and Oculus

versions are similar in terms of learning, enjoyment, aesthetics and controls. Since VR facilitates

an immersive environment, the participants who played the game with the Oculus VR headset

were less aware of their surroundings thus implying that they were more immersed in the game.

However, fewer participants from the Oculus group felt that time slowed down, and the game

was relatively short for them compared to those who played the Desktop version. From the 7-

point Likert scale (Figure 4.1.b), it can be implied that participants who played in the Oculus

environment were more focused and interacted with the in-game objects more as compared to the

Desktop version. Additionally, it also implies that participants were more immersed in the

Oculus version as they expressed that they were experiencing events rather than doing them.

The quantitative data of total time spent on each object was obtained from the video transcript.

The data from table 4.2.a indicated that participants from the Oculus group spent more time on

the planets in contrast with the participants from the Desktop group. To further investigate the

23

difference between the time spent, a non-parametric test was performed. The result of the

independent 2-group Mann-Whitney U Test was not able to reject the null hypothesis meaning

that we cannot conclude that there is a significant difference between the time spent on objects

between the two groups.

5.2 Qualitative data

The results from the sentiment analysis of the descriptions gives an insight about the elements of

the game and their how they affect the players. Some of the participants from the Desktop group

found it difficult to concentrate on the game due to the overwhelming information and facts

about the planets. This might also be the reason that they got bored while playing. Two of the

participants also found the in-game music to be distracting while concentrating on the

information about the planets and stars. However, most of the participants expressed enjoyment

while playing the game. In the case of the Oculus version, most participants were fearful of the

stars which appeared suddenly in their vision. Few of the participants felt uneasy and motion

sickness due to the movement in VR, which is a common disadvantage of using VR for any

application. Most of the participants who were alright with VR were excited and impressed by

the appealing graphics and textures of planets.

The results from the cognition categories indicated that participants from both the groups were

almost equally engaged in learning from the game. This reinforces the results of learning

obtained from the Likert scales. From the Affect and Experience categories, the participants who

played the Oculus version were more emotionally involved and affected by the game and talked

that they were experiencing rather than just playing the game than those who played on Desktop

24

computer. It was also observed from the Evaluation and Game Design categories that the

participants who played the desktop version were more analytical about the game and were

reviewing the game mechanics, whereas the Visual Design category indicated that the

participants from the Oculus version talked more about and paid attention to the graphics and

aesthetics like textures, colors, shapes, and sizes of planets and stars.

It was observed that the two different modalities mediate the player's interaction with the game

content differently. This motivates further research into which modalities are appropriate for

different kinds of learning. It would seem, tentatively, from this exploratory study, that VR

supports strong affective engagement with visually rich content, while a Desktop experience

supports a more detached, cognitive and reflective examination of the content. Depending on the

learning goals of a particular context, either modality may be more appropriate.

.

25

6. CONCLUSION

It can be concluded that Virtual Reality seems to be stronger in some areas, and the Desktop 3D

stronger in others, and in some areas, they are virtually identical. The results obtained from the

quantitative and qualitative data reinforced each other to suggest that Virtual Reality is better in

games or applications which are centralized on artistic and immersive content that encourages

the user to be more emotionally involved. In a case such as a phobia therapy, where strong

affective engagement is a more important function, the Oculus VR headset might be better.

However, in the learning area, which is the goal of this particular application, Desktop 3D may

in fact be better as users tend to be more focused and intellectual while comprehending

information in that modality. In future studies, analyzing data from a larger population of both

the groups may give an accurate insight and we might be able to observe significant differences

between Virtual Reality and Desktop 3D as a medium for games.

26

7. REFERENCES

1. Virvou, M., Katsionis, G. (2008, January). On the usability and likeability of virtual

reality games for education: The case of VR-ENGAGE. Computers & Education,

Volume 50, Issue 1, Pages 154-178. Retrieved from

https://doi.org/10.1016/j.compedu.2006.04.004

2. Papert, S. (1980). Mindstorms: Children, Computers and Powerful Ideas. New York:

Basic Books. Retrieved from https://dl.acm.org/citation.cfm?id=1095592

3. Mills, S., Noyes, J. (1999, April). Virtual reality: an overview of User-related Design

Issues: Revised Paper for Special Issue on “Virtual reality: User Issues” in Interacting

with Computers, May 1998. Interacting with Computers, Volume 11, Issue 4, Pages 375–

386. Retrieved from https://doi.org/10.1016/S0953-5438(98)00057-5

4. Tuveri, E., Macis, L., Sorrentino, F., Spano, L. D., Scateni, R. (2016, June). Fitmersive

Games: Fitness Gamification through Immersive VR. Proceedings of the International

Working Conference on Advanced Visual Interfaces, Pages 212-215. Retrieved from

https://dl.acm.org/citation.cfm?doid=2909132.2909287

5. Huang, H., Rauch, U., Liaw, S. (2010, November). Investigating learners’ attitudes

toward virtual reality learning environments: Based on a constructivist approach.

Computers & Education, Volume 55, Issue 3, Pages 1171-1182. Retrieved from

https://doi.org/10.1016/j.compedu.2010.05.014

6. Merchant, Z., Goetz, E. T., Cifuentes, L., Keeney-Kennicutt, W., Davis, T. J. (2014,

January). Effectiveness of virtual reality-based instruction on students’ learning outcomes

in K-12 and higher education: A meta-analysis. Computers & Education, Volume 70,

Pages 29-40. Retrieved from https://doi.org/10.1016/j.compedu.2013.07.033

7. Josman, N., Ben-Chaim, H. M., Friedrich, S., Weiss, P. L. (2008, January). Effectiveness

of virtual reality for teaching street-crossing skills to children and adolescents with

autism. International Journal of Disability Development and Education. DOI:

10.1515/IJDHD.2008.7.1.49. Retrieved from

https://www.researchgate.net/publication/236658997

27

8. Lee, E. A., Wong, K. W., Fung, C. C. (2010, December). How does desktop virtual

reality enhance learning outcomes? A structural equation modeling approach. Computers

& Education, Volume 55, Issue 4, Pages 1424-1442. Retrieved from

https://doi.org/10.1016/j.compedu.2010.06.006

9. Telegraph Reporters (March 2018). The best virtual reality headsets you can buy.

Retrieved from https://www.telegraph.co.uk/technology/0/best-virtual-reality-headsets-

can-buy/

10. DrashVR LLC. (2016). Titans of Space 2.0 (Version 4) [Computer application software].

Retrieved from http://store.steampowered.com/app/468820/Titans_of_Space_20/

11. Krijn, M., Emmelkamp, P. M.G., Biemond, R., De Wilde de Ligny, C., Schuemie, M. J.,

Van der Mast, C. A.P.G. (February 2004). Treatment of acrophobia in virtual reality: The

role of immersion and presence. Behaviour Research and Therapy,Volume 42, Issue 2,

Pages 229–239. Retrieved from https://doi.org/10.1016/S0005-7967(03)00139-6

12. Silge, J., Robinson, D. (September 2018). Text Mining with R, Chapter 2: Sentiment

analysis with tidy data. Retrieved from https://www.tidytextmining.com/sentiment.html

13. TechSmith (2018). Morae Components and Features [Web article]. Retrieved from

https://www.techsmith.com/morae-features.html

28

9. APPENDIX A:

Survey:

The participants were given an identity code to note their responses:

The 5-point Likert scale:

29

Individual questions with a 7-Point Likert scale:

30

Remaining 7-point Likert scale questions with the description section to note their experience: