Audition, The Game:Assessing the possibilities of speech as a non-trivial

gameplay element in video gamesT.J. Lavender

IAT 881 Bioaffective ComputingSN 200113129April 25, 2011

tjlavend@sfu.ca

ABSTRACTThis paper describes Audition, The Game, a prototype video game created to explore the possible use of speech as a challenge element in video games. In the game, players needed to speak certain phrases in order to win. Galvanic skin response was recorded during game play to determine whether psychophysiological measurements could be used to evaluate speech as a nontrivial game play element. Initial results were promising enough to warrant further development and research.

Author KeywordsVideo games, speech, GSR, affective computing, psychophysiological measurement.

ACM Classification KeywordsK8.0 Personal Computing. Games.

INTRODUCTIONSound is widely used in video games. Generally, it is used to create or enhance moods, to add realism or create certain effects, for example, the sound of bullets in a first person shooter, crowd noises in a sports game, characters speaking in a role playing game, miscellaneous sound effects in arcade games, ambient spooky music in an adventure game. Few if any videogames do not use sound, whether it be a musical soundtrack, sound effects or NPC speech (Collins, 2008). However, sound is rarely used as an active game play element, where it can have either a principal or secondary role in the game outcome.One reason sound, and especially speech, does not play an active role in game play is the difficulty of evaluation. For example, consider a game where the player had to give a password to get by a guard. How would the game engine evaluate if the password was said correctly, given the large variation in speech (accent, timbre, volume, duration) among individuals?     It is possible that over time, voice recognition will improve to a point where it could be used as a non-trivial game play element, but at present this is not practical [1]. 

In the shorter term, one possibility is to incorporate psychophysiological measurement, such as GSR or EEG to evaluate the player's physiological responses while

speaking. This would be cumbersome at present, requiring intrusive and awkward measuring equipment such as a headset or finger sensors [3]. However, a new generation of game controllers incorporating rudimentary GSR measurement is already on the horizon and will shortly be available [2].Before this instrumentation can be used to evaluate speech for game play, it will be necessary to determine first, whether speech would be an engaging challenge element for game players and second, whether speech can be reliably evaluated through psychophysiological measurements such as GSR. To date, little research has been done in this area, although some investigation is underway into other uses of psychophysiological measurement in video games, such as the mapping between EEG readings and the emotions of game players [3] and between anxiety level and challenge [5].

IMPLICATIONS FOR VIDEO GAME DEVELOPMENTResearch into the use of speech as a non-trivial game play element would increase possibilities for game developers. Game engines that sense frustration, fear, boredom or other emotions in the voice of the player could adjust the game play accordingly. And new challenges could be developed based on vocal dexterity, correctly recalling and reciting phrases, etc. These elements would be useful in standard

1

Figure 1. Audition, The Game screenshot.

entertainment games and also in serious games, such as games developed to assist people with speech impairments.

IMPLICATIONS FOR AFFECTIVE COMPUTINGThis research may also have implications for the field of affective computing. As Picard [4] notes, characterizing affect in speech is difficult, as is finding an accurate map between emotions and speech patterns. Some research points to a correlation between fear and speech that is faster, louder and more enunciated, but many questions remain about the vocal parameters of affect [4]. If a correlation is found between GSR (or other psychophysiological measurements such as heart rate) and speech difficulty and anxiety in Audition, the Game, it may be possible to use measurement of these states to determine whether an individual is struggling with speech, is under stress or experiencing strong emotions or conversely whether speech difficulty is affecting the person in other ways. 

THE GAMEAudition, The Game (Figure 1, and Appendix) is an attempt to explore the potential of using speech as a principal challenge element in videogames. The player controls a character (from a 3rd person point of view) who must pass an audition or abandon her dream of becoming an actor. The character must first get to the audition riding a skateboard through traffic that she guides using either a mouse or trackpad with her preferred hand. She then must flawlessly recite a tongue-twisting passage within a certain space of time (Figure 2). If she succeeds, she then has to proceed to a second, harder, audition, again riding her board through traffic. If she succeeds in the second audition, she then has to go to the final audition, which again is more difficult. Passing this audition will land her the starring role in the movie and win the game.An added element of game play is the need to keep calm while reciting the tongue-twisters. Calmness is measured through Galvanic Skin Response (GSR) sensors. This

element was added to the game story line in order to justify to the player the need for the GSR instrumentation.

MEASUREMENT AND EVALUATIONDuring game play, GSR instrumentation will be attached to fingers on the player's non-game playing hand and the GSR readings will be recorded as the player recites the three tongue twisters. The readings will be analyzed later for possible correlation between GSR and speech difficulty.

PROTOTYPE OBJECTIVESThe prototype was developed to evaluate the potential for using speech as a game element. Would the tongue twisters be challenging enough to engage video game players? Would the GSR stress measurement impact the main game element (reciting the tongue twisters)? What effect, if any, would the skateboard challenge have on the GSR and the speech aspects of the game?The game prototype has been tested with a small group of volunteers. Game play has been judged as satisfactory, with both the skateboard element and the tongue twister auditions providing enough of a challenge to sustain interest (Figure 3). GSR readings showed GSR rising in anticipation of the audition and during the initial speaking of the tongue twisters and then leveling out. However, the sample (n = 4) was too small to draw any definite conclusions.

LIMITATIONSDue to technical issues and time limitations, it is currently in three separate parts, Audition 1, Audition 2, Audition 3. The GSR monitoring is not incorporated into the game, but is done on a separate computer. The skateboarding game element is repetitive.

GSR was chosen for the psychophysiological measurement due to accessibility and ease of use. However, it would not be useful in a real game setting as it is a relative measure and a baseline would need to be established for each individual player [5]. It has also been suggested integrating

2

Figure 3. Both the skateboarding and audition speaking game play elements were engaging enough to

sustain interest in the prototype demonstration.

Figure 2. Success in the game is dependent on correctly reciting increasingly complex tongue twisters in a limited

period of time.

the GSR reading into the actual game and displaying it on the screen in real time would enhance the game play.

FUTURE DIRECTIONSBased on feedback during the prototype demonstration, the next version of the game will combine the three game stages into a single game. The graphics will be improved and the skateboarding element of the gameplay will be more varied. The game will be tested with a larger group of people and both heart rate and GSR readings will be recorded and displayed on the screen in real time. Players will also be asked to evaluate the game play through a questionnaire and a semi-structured post-experiment interview. Should an analysis of the GSR and heart rate data show a correlation with the speech elements of the game, further research would be warranted.

ACKNOWLEDGMENTSAudition, The Game is based on a tutorial game created by Todd Perkins for lynda.com. The characters, voices, and machinima were rendered through the services of xtranormal.com. Sound effects are courtesy of partnersinrhyme.com. Thanks are due to Diane Gromala and the students in her IAT 881 Bioaffective Computing class at Simon Fraser University’s School of Interactive Arts and Technology for suggestions and inspiration.

REFERENCES1. Anusuya, M. and Katti, S. 2009. Speech recognition by

machine: a review. International Journal of Computer Science and Information Security. 6, 3, 181-205.

2. Aron, J. 2011. Play it again, but this time with feeling. New Scientist.

3. Nacke, L. E. and Lindley, C. A. 2010. Affective Ludology, Flow and Immersion in a First- Person Shooter: Measurement of Player Experience. eprint arXiv. 1004, 248.

4. Picard, R. W. 1997 Affective Computing. Massachusetts Institute of Technology.

5. Rani, P., Sarkar, N., and Liu, C. 2005. Maintaining optimal challenge in computer games through real-time physiological feedback. Proceedings of the 11th International Conference on Human Computer Interaction. 184–192.

3

APPENDIX: AUDITION, THE GAME FLOWCHART

4

intro(intro.flv)

first skateboard

ride

success

first audition(audition1.flv)

failure

successfailure(audition1_f.flv)

failure(audition3_f.flv)

failure(audition2_f.flv)

second skateboard

ride

third skateboard

ride

second audition

(audition2.flv)

third audition(audition3.flv)

game lost(finale_f.flv)

game won (finale_w.flv)

success

failure

failure

success(audition1_s.flv)

success(audition2_s.flv)

success(audition3_s.flv)

Audition: The Gameflowchart