Exploring the Effects ofEncouragement in Educational Games

Dominic KaoMIT77 Massachusetts AveCambridge, MA 02139 USAdkao@mit.edu

D. Fox HarrellMIT77 Massachusetts AveCambridge, MA 02139 USAfox.harrell@mit.edu

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s). Copyright is held by theauthor/owner(s).CHI’16 Extended Abstracts, May 7–12, 2016, San Jose, CA, USA.ACM 978-1-4503-4082-3/16/05.http://dx.doi.org/10.1145/2851581.2892335

AbstractEncouragement (e.g., “You’re doing well”) given at regu-lar intervals improves performance in a variety of sportingdomains [9, 1, 6]. This improvement is regardless of theactual performance of participants. However, it has notbeen studied how this type of encouragement can affectplayers of video games. In the current study (N = 662), welook at the following encouragement conditions: (1) Positive(e.g., “You’re doing good”), (2) Negative (e.g., “You’re doingbadly”), (3) Neutral (e.g., “You’re doing average”), and (4)None. Via the Game Experience Questionnaire (GEQ) [12],participants in the Neutral condition had significantly im-proved flow, immersion, and affect than participants in theNone condition. Moreover, participants in both the Positiveand Neutral conditions had the highest overall GEQ ratings.These findings are directly relevant to educational games.

Author KeywordsEncouragement; Educational Games; Feedback

ACM Classification KeywordsK.8.0 [Personal Computing]: General – Games

IntroductionSimple phrases of encouragement (e.g.,”You’re doing well”)delivered at 30-second intervals, significantly improves per-formance in walking distance [9]. Numerous studies have

Late-Breaking Work: Games & Playful Interaction #chi4good, CHI 2016, San Jose, CA, USA

1906

reproduced similar results in a variety of strength and en-durance domains [36, 22, 6, 1]. However, few studies onthese types of interventions have been studied in games.O’Rourke et. al found that encouraging the development ofa growth mindset, or the belief that intelligence is malleable,increases player perseverance [29, 8]. To the best of ourknowledge, however, no studies have attempted to studythis simple encouragement inside games.

Encouragement is different from feedback, in that it doesn’tnecessarily encode information about performance [31, 20,7, 23]. Our experiment follows a similar model to previousones on encouragement [9, 36, 22, 6, 1]. The encourage-ment is: 1) Always the same valence depending on con-dition, 2) Speaks to the task at hand and not the learner,and 3) Dispensed at regular time intervals [35, 24, 5]. Ourgoal is to study how game experience is affected by encour-agement, and whether it is positively affected relative to noencouragement at all.

Figure 1: Level 1 in Mazzyintroduces the basic gamemechanics.

Figure 2: Level 6 introduceslooping.

The GameThe experiment takes place in a STEM learning gamecalled Mazzy [16]. Mazzy is a game in which players solvemazes by creating short computer programs. In total, thereare 12 levels in this version of Mazzy. Levels 1-5 requireonly basic commands. Levels 6-9 require using loops. Lev-els 10-12 require using all preceding commands in additionto conditionals. See Figures 1 and 2. Mazzy has been usedpreviously as an experimental testbed for evaluating the im-pacts of avatar type on performance and engagement in aneducational game [19, 17, 15, 18, 14].

MethodsOur experiment aims to compare four encouragement con-ditions: (1) Positive, (2) Negative, (3) Neutral and (4) None.The goal is to see if participants in these conditions have

Condition Sentence ScorePositive You’re doing well 3.10Positive Don’t give up! 2.44Positive You’re almost there 2.29Negative You work poorly -3.43Negative You’re on the wrong track -2.12Negative You’re still far away -1.53Neutral You are doing standard work 0.08Neutral You’re doing average 0.03Neutral You’re doing typically 0.01

Table 1: Example sentences

different game performance and game experience as mea-sured by the GEQ.

Creating SentencesIn designing the sentences for each condition, 150 sen-tences were drafted (50 for each of positive, negative, andneutral conditions). These were developed based on pre-vious encouragement studies [9, 2, 33]. We then recruited103 U.S. participants to rate the sentences on a scale of-5:Very Negative to 5:Very Positive. Intraclass correlationon the questions was ICC = 0.99 (two-way random, averagemeasures [34]), indicating high agreement.

20 sentences were then randomly selected for each con-dition. In doing so, each randomly selected positive sen-tence was matched to the negative sentence with the clos-est opposite numeric valence score. The average wordsper sentence did not differ significantly between any of theconditions, p > .05. The final average valence scores forthe positive sentences was 2.75, for the negative sentences-2.75, and for the neutral sentences 0.00. See Table 1 forexamples.

Late-Breaking Work: Games & Playful Interaction #chi4good, CHI 2016, San Jose, CA, USA

1907

ConditionsThe four encouragement conditions we tested were:

a. Positiveb. Negativec. Neutrald. None

Sentences appeared centered at the bottom of the screenin 28 px font. The words appeared on a 46 px high semi-transparent black bar. Procedure, instructions, gameplay,were all exactly identical across all conditions, only the sen-tences appearing were different. One randomly chosensentence was shown at 30 second intervals. Each sentencewas displayed for 15 seconds. In the None condition, theblack bar was still displayed, but no text was shown. SeeFigures 3, 4, 5, and 6.

Figure 3: Positive condition.

Figure 4: Negative condition.

Figure 5: Neutral condition.

Figure 6: None condition.

Quantitative and Qualitative MeasuresFor performance, we looked at number of levels completedby players. For measuring game experience, we use theGEQ [12].

Participants662 participants were recruited through Mechanical Turk.The data set consisted of 51.6% male, and 48.4% femaleparticipants. Participants self-identified their races/ethnicitiesas white (80.5%), black or African American (9%), Chi-nese (2.3%), Asian Indian (1.2%), Filipino (0.9%), Korean(0.8%), American Indian (0.6%), Japanese (0.5%), andother (4.1%). Participants were between the ages of 18and 78 (M = 32.3, SD = 9.7), and were all from the UnitedStates. Participants played the game a single time for anaverage length of 22.9 minutes. Participants were reim-bursed $1.50 to participate in this experiment.

DesignA between-subjects design was used: encouragement va-lence was the between-subject factor. Participants wererandomly assigned to a condition.

ProtocolPrior to starting the game, players were informed that theycould exit the game at any time via a red button in the cor-ner of the screen. When participants were done playing(either by exiting early, or by finishing all 12 levels), partic-ipants returned to the experiment instructions, which thenprompted them with the GEQ and then a demographics sur-vey.

AnalysisPlayer responses were analyzed using multivariate analysisof variance (MANOVA) in SPSS. The dependent variableswere- GEQ items; and the independent variable was- en-couragement (positive, negative, neutral, or none). All thedependent variables are continuous variables. The inde-pendent variable encouragement (i.e., 0 = positive, 1 = neg-ative, 2 = neutral, 3 = none) was a quadchotomous variable.To detect the significant differences between encourage-ment conditions, we utilized one-way MANOVA. We alsoused one-way ANOVA on the variable- levels completed.These results are reported as significant when p<0.05 (two-tailed). Before running MANOVA, all the variables includedin the analyses were checked, and there were 17 outliersdetected [11]. These 17 outliers were excluded from furtheranalysis. Prior to running our MANOVA, we checked as-sumption of homogeneity of variance by the test of Levene’sTest of Equality of Error Variances; and the assumption wasmet by the data (p>.05).

Late-Breaking Work: Games & Playful Interaction #chi4good, CHI 2016, San Jose, CA, USA

1908

ResultsAggregateThe one-way ANOVA found no significant effect of encour-agement valence on levels completed, F(3, 641) = 1.51, p =0.21 (see Table 2).

A MANOVA revealed a statistically significant difference inGEQ responses based on the participant’s encouragementvalence, F(126, 1799) = 1.44, p <.005; Wilk’s λ = 0.750,partial η2 = .09. See Figure 7.

Figure 7: Game Experience Questionnaire (GEQ) responses.

Valence N Mean SDPositive 147 7.69 2.79Negative 161 7.16 2.85Neutral 151 7.78 2.78None 186 7.49 2.84

Table 2: Overall level completion statistics.

Univariate testing found the effect to be significant for thefollowing items:

• Flow:“I felt completely absorbed”, p <0.001.ad

“I was deeply concentrating on the game”, p <0.05.a

• Immersion:“I was interested in the game’s story”, p <0.01.a

“I felt imaginative”, p <0.005.ad

“I felt that I could explore things”, p <0.005.a

“I found it impressive”, p <0.005.a

“It felt like a rich experience”, p <0.005.a

• Competence:“I felt skillful”, p <0.05.a

“I felt strong”, p <0.05.a

“I was good at it”, p <0.01.b

“I felt successful”, p <0.001.abc

“I was fast at reaching the game’s targets”, p <0.01.ab

• Challenge:“I felt that I was learning”, p <0.05.a

“I felt stimulated”, p <0.005.ab

“I felt time pressure”, p <0.05.c

• Tension:“I felt tense”, p <0.05.bc

“I felt restless”, p <0.01.a

“I felt annoyed”, p <0.001.abc

“I felt irritable”, p <0.005.ab

“I felt frustrated”, p <0.05.ac

Late-Breaking Work: Games & Playful Interaction #chi4good, CHI 2016, San Jose, CA, USA

1909

“I felt pressured”, p <0.005.abc

• Positive Affect:“I felt content”, p <0.001.abc

“I felt happy”, p <0.001.a

“I felt good”, p <0.001.abc

“I enjoyed it”, p <0.001.abd

“I thought it was fun”, p <0.001.abd

• Negative Affect:“I thought about other things”, p <0.01. a

“I found it tiresome”, p <0.01. ad

“I felt bored”, p <0.001.abd

“I was distracted”, p <0.05.a

“I was bored by the story”, p <0.01.a

“It gave me a bad mood”, p <0.05.c

In order to compare the effects of encouragement type onthese measures, we additionally calculated posthoc com-parisons (Tukey HSD) between all conditions. Superscriptsdenote cases when Neutral outperforms Negative (a), Pos-itive outperforms Negative (b), None outperforms Negative(c), and Neutral outperforms None (d). We note that thesheer consistency across all questions indicates an order-ing (i.e., Figure 7).

Why Does Neutral Outperform Positive?Participants in the Neutral condition have the highest GEQratings (aside from competence). Investigating, we lookedat responses to “How did you feel about the [encourage-ment] text in the game?”. Words most often used to de-scribe the encouragement text in the Positive conditionwere “encouraging” and “positive”. Participant No. 27 de-scribed it as:

“It was encouraging. Made me smile a bit eventhough I knew I was doing terrible at the game.”

Some participants (13%) found the positive encouragementtext less helpful. Participant No. 98 said:

“I liked that the feedback was encouraging, butit seemed “fake” in the sense that no matterwhat I did, I was going to receive positive feed-back. That cheapened it a bit.”

On the other hand, players used words like “indifferent” and“encouraging” to describe the words in the Neutral condi-tion. Participant No. 119 said:

“Sort of helpful. It made me feel a little betterknowing I was at least average, when I figured Iwas totally sucking.”

Participant No. 115 said:

“I felt like it brought me down a little and addeda little bit of pressure, yet I could ignore it easilyhad I wanted too [sic].”

Participants No. 22 and No. 56 suggested that “it [the neu-tral text] pushed me to work harder” and “it [the neutral text]was humorously neutral”. From these responses, it’s clearthat the impacts of encouragement were variable. Virtuallyall participants found the positive encouragement text to behelpful early on. However, many players that progressedpast half-way (Level 7 and onwards) found the text to be“fake”. For these players, the following three things werehappening simultaneously: 1) The game was becomingharder, 2) The participants were experiencing frustration,and 3) The positive text combined with the participant’sfrustration served only to further increase frustration.

Late-Breaking Work: Games & Playful Interaction #chi4good, CHI 2016, San Jose, CA, USA

1910

On the other hand, participants in the neutral encourage-ment condition did not have responses that varied by howfar they had progressed in the game. They expressed alevel of indifference; a few participants explicitly stated theneutral text had a motivating effect, e.g., to be better thanaverage. Contrary to the participants in the positive encour-agement condition, participants in the neutral encourage-ment condition never felt that the text was fake.

DiscussionWe have seen from our results that players with neutraland positive encouragement had the highest engagement.Our measurement instrument was the GEQ. The GEQ wasused for its multiple subscales which assess different com-ponents of the player experience, and it is both a widelyused and recognized instrument [28]. Although the GEQwas adequate for measuring engagement, there are a num-ber of viable alternatives [27]. Instruments such as the BigFive Inventory (BFI) [13] could further shed light on themoderating effects of player personality.

Research on Feedback Interventions (FIs) have shown thatpredicting the effect of any given feedback is contingent ona wide array of factors: personality, feedback type (verbal,etc.), frequency of feedback, task complexity, task novelty,type of task (physical, etc.), etc. [20, 4, 25, 35, 26, 32, 21].Therefore, researchers should be wary of prescribing gen-eral guidelines regarding encouragement.

Keeping in mind the numerous contextual moderators, ourresults suggest that encouragement can improve game ex-perience. Even in a setting where the encouragement wasnot directly connected in any way to the gameplay, resultsshowed significant increases in flow, immersion, positive af-fect, etc. Positive encouragement appeared to benefit play-ers most when the game was easy; those benefits tapered

as the game progressively became harder (the results areconsistent with work in which insincere praise has a neg-ative effect [10, 30]). Encouragement models that bettermatch player performance, e.g., acknowledging the player’sstruggles in an encouraging tone, could yield greater bene-fits.

ConclusionIn this paper, we have explored the effects of different typesof encouragement. We have shown that encouragement(relative to no encouragement) can improve the game ex-perience of players. This is consistent with other work onencouragement [9, 1, 36, 22, 6]. While being mindful of thehighly contextual nature of this topic, educational gamescan consider encouragement as a means to improve gameexperience. Better engaging learners is one route towardscreating more meaningful learning experiences [3].

Future WorkOne possible direction for future work is to study the con-textual determinants of how participants are affected byencouragement, e.g., player characteristics such as age,personality, etc. Studying how to increase the effectivenessof different forms of encouragement (better customizing textto player action, using a virtual avatar whose facial expres-sions match the encouragement valence, etc.) could alsobe interesting.

AcknowledgmentsWe would like to thank the anonymous reviewers for theirvaluable feedback. This work is supported by NSF STEM+CGrant 1542970 and a Natural Sciences and EngineeringResearch Council of Canada (NSERC) fellowship.

Late-Breaking Work: Games & Playful Interaction #chi4good, CHI 2016, San Jose, CA, USA

1911

References[1] Joseph L Andreacci, Linda M LeMura, Steven L Co-

hen, Ethan a Urbansky, Sara a Chelland, and Serge PVon Duvillard. 2002. The effects of frequency of en-couragement on performance during maximal exercisetesting. Journal of sports sciences 20, 4 (2002), 345–352. DOI:http://dx.doi.org/10.1080/026404102753576125

[2] Boaz M Ben-David, Pascal H H M van Lieshout,and Talia Leszcz. 2011. A resource of validated af-fective and neutral sentences to assess identifica-tion of emotion in spoken language after a brain in-jury. Brain injury : [BI] 25, 2 (2011), 206–20. DOI:http://dx.doi.org/10.3109/02699052.2010.536197

[3] Phyllis C. Blumenfeld, Toni M Kempler, and Joseph S.Krajcik. 2005. Motivation and Cognitive Engagement inLearning Environments. In The Cambridge Handbookof the Learning Sciences. 475–488. DOI:http://dx.doi.org/10.1017/CBO9780511816833.029

[4] E. Brummelman, S. Thomaes, B. Orobio de Castro, G.Overbeek, and B. J. Bushman. 2014a. "That’s Not JustBeautiful–That’s Incredibly Beautiful!": The AdverseImpact of Inflated Praise on Children With Low Self-Esteem. Psychological Science 25, 3 (2014), 728–735. DOI:http://dx.doi.org/10.1177/0956797613514251

[5] Eddie Brummelman, Sander Thomaes, Geertjan Over-beek, Bram Orobio de Castro, Marcel a van den Hout,and Brad J Bushman. 2014b. On feeding thosehungry for praise: Person praise backfires in chil-dren with low self-esteem. Journal of Experimen-tal Psychology: General 143, 1 (2014), 9–14. DOI:http://dx.doi.org/10.1037/a0031917

[6] José Marinho Dias Neto, Fernanda Borges Silva, Ar-tur Luis Bessa De Oliveira, Natália Lopes Couto, Es-télio Henrique Martins Dantas, and Maria AparecidaDe Luca Nascimento. 2015. Effects of verbal en-

couragement on performance of the multistage 20 mshuttle run. Acta Scientiarum. Health Sciences 37, 1(2015), 25. DOI:http://dx.doi.org/10.4025/actascihealthsci.v37i1.23262

[7] Sadler D.R. 1989. Formative Assessment and thedesign of instructional systems. Instructional Sci-ence 18 (1989), 119–144. DOI:http://dx.doi.org/10.1007/BF00117714

[8] Carol Dweck. 2006. Mindset: The New Psychologyof Success. Random House Publishing Group. 276pages. http://books.google.com/books?hl=en

[9] G H Guyatt, S O Pugsley, M J Sullivan, P J Thomp-son, L Berman, N L Jones, E L Fallen, and D W Taylor.1984. Effect of encouragement on walking test per-formance. Thorax 39, 11 (1984), 818–822. DOI:http://dx.doi.org/10.1136/thx.39.11.818

[10] Jennifer Henderlong and Mark R Lepper. 2002. Theeffects of praise on children’s intrinsic motivation: areview and synthesis. Psychological bulletin 128,5 (2002), 774–795. DOI:http://dx.doi.org/10.1037/0033-2909.128.5.774

[11] David C. Hoaglin and Boris Iglewicz. 1987. Fine-Tuning Some Resistant Rules for Outlier Labeling.J. Amer. Statist. Assoc. 82, 400 (1987), 1147–1149.DOI:http://dx.doi.org/10.1080/01621459.1987.10478551

[12] W IJsselsteijn, Y De Kort, K Poels, A Jurgelionis,and Francesco Bellotti. 2007. Characterising andMeasuring User Experiences in Digital Games. In-ternational Conference on Advances in ComputerEntertainment Technology 620 (2007), 1–4. DOI:http://dx.doi.org/10.1007/978-1-60761-580-4

[13] Op P John and S Srivastava. 1999. The Big Fivetrait taxonomy: History, measurement, and theoret-ical perspectives. Handbook of personality: The-ory and research 2, 510 (1999), 102–138. DOI:http://dx.doi.org/citeulike-article-id:3488537

Late-Breaking Work: Games & Playful Interaction #chi4good, CHI 2016, San Jose, CA, USA

1912

[14] Dominic Kao and D. Fox Harrell. 2015a. Exploringconstruction, play, use of virtual identities in STEMlearning. Jean Piaget Society Annual Conference(2015).

[15] Dominic Kao and D. Fox Harrell. 2015b. Exploring theImpact of Role Model Avatars on Game Experience inEducational Games. The ACM SIGCHI Annual Sym-posium on Computer-Human Interaction in Play (CHIPLAY) (2015).

[16] Dominic Kao and D. Fox Harrell. 2015c. Mazzy: ASTEM Learning Game. Foundations of Digital Games(2015).

[17] Dominic Kao and D. Fox Harrell. 2015d. Toward AvatarModels to Enhance Performance and Engagement inEducational Games. IEEE Computational Intelligencein Games (2015).

[18] Dominic Kao and D. Fox Harrell. 2015e. Toward Evalu-ating the Impacts of Virtual Identities on STEM Learn-ing. Foundations of Digital Games (2015).

[19] Dominic Kao and D. Fox Harrell. 2016. Toward Un-derstanding the Impacts of Role Model Avatars on En-gagement in Computer Science Learning. The annualmeeting of the American Educational Research Asso-ciation (AERA) (2016).

[20] Avraham N Kluger and Angelo DeNisi. 1996. Theeffects of feedback interventions on performance: ahistorical review, a meta-analysis and a preliminaryfeedback intervention theory. . Psychological bulletin119, 2 (1996), 254–284. DOI:http://dx.doi.org/10.1037/0033-2909.119.2.254

[21] Björn Krenn, Sabine Würth, and Andreas Hergovich.2013. The impact of feedback on goal setting and taskperformance: Testing the feedback intervention theory.Swiss Journal of Psychology (2013).

[22] R. J. Moffatt, L. F. Chitwood, and K. D. Biggerstaff.1994. The influence of verbal encouragement during

assessment of maximal oxygen uptake. Journal ofSports Medicine and Physical Fitness 34, 1 (1994),45–49.

[23] Elizabeth K Molloy and David Boud. 2014. Hand-book of Research on Educational Communica-tions and Technology. (2014), 413–424. DOI:http://dx.doi.org/10.1007/978-1-4614-3185-5

[24] C M Mueller and C S Dweck. 1998. Praise for intelli-gence can undermine children’s motivation and per-formance. Journal of personality and social psychol-ogy 75, 1 (1998), 33–52. DOI:http://dx.doi.org/10.1037/0022-3514.75.1.33

[25] Jonathan Mumm and Bilge Mutlu. 2011. Designingmotivational agents: The role of praise, social compar-ison, and embodiment in computer feedback. Com-puters in Human Behavior 27, 5 (2011), 1643–1650.DOI:http://dx.doi.org/10.1016/j.chb.2011.02.002

[26] Susanne Narciss. 2013. Designing and evaluatingtutoring feedback strategies for digital learning environ-ments on the basis of the interactive tutoring feedbackmodel. Digital Education Review 23, 1 (2013), 7–26.

[27] A. Imran Nordin, Alena Denisova, and Paul Cairns.2014. Too Many Questionnaires: Measuring PlayerExperience Whilst Playing Digital Games. SeventhYork Doctoral Symposium on Computer Science &Electronics 69 (2014).

[28] K. L. Norman. 2013. GEQ (Game Engage-ment/Experience Questionnaire): A Review of TwoPapers. Interacting with Computers 25, 4 (mar 2013),278–283. DOI:http://dx.doi.org/10.1093/iwc/iwt009

[29] E O’Rourke and Kyla Haimovitz. 2014. Brain points:a growth mindset incentive structure boosts persis-tence in an educational game. Proceedings of the32nd annual ACM conference on Human factors incomputing systems - CHI ’14 (2014), 3339–3348.http://dl.acm.org/citation.cfm?id=2557157

Late-Breaking Work: Games & Playful Interaction #chi4good, CHI 2016, San Jose, CA, USA

1913

[30] Joachim S Prook, Dirk P Janssen, and StefanoGualeni. 2015. The Negative Effects of Praise andFlattery. FDG (2015).

[31] Arkalgud Ramaprasad. 1983. On the definition offeedback. Behavioral Science 28, 1 (1983), 4–13.DOI:http://dx.doi.org/10.1002/bs.3830280103

[32] Ido Roll, Vincent Aleven, Bruce M. McLaren, and Ken-neth R. Koedinger. 2011. Improving students’ help-seeking skills using metacognitive feedback in an in-telligent tutoring system. Learning and Instruction21, 2 (2011), 267–280. DOI:http://dx.doi.org/10.1016/j.learninstruc.2010.07.004

[33] Jeff B Russ, Ruben C Gur, and Warren B Bilker. 2008.Validation of affective and neutral sentence content for

prosodic testing. Behavior research methods 40, 4(2008), 935–9. DOI:http://dx.doi.org/10.3758/BRM.40.4.935

[34] P E Shrout and J L Fleiss. 1979. Intraclass cor-relations: uses in assessing rater reliability. Psy-chological bulletin 86, 2 (1979), 420–428. DOI:http://dx.doi.org/10.1037/0033-2909.86.2.420

[35] V. J. Shute. 2008. Focus on Formative Feedback.Review of Educational Research 78, 1 (2008), 153–189. DOI:http://dx.doi.org/10.3102/0034654307313795

[36] J B Wise, A E Posner, and G L Walker. 2004. Verbalmessages strengthen bench press efficacy. Journalof Strength & Conditioning Research 18, 1 (2004), 26–29.

Late-Breaking Work: Games & Playful Interaction #chi4good, CHI 2016, San Jose, CA, USA

1914