1

Sound and music in serious games

Anja Volk

Sound and Music Technology,Nov 23, 2017

2

Recap

g Sound and music for interactivity and immersion in games.• How to achieve interactive music? (Michael Sweet):

• Method 1: Vertical Remixing (Layering)• Method 2: Horizontal Re-sequencing

• Immersion: 3 different aspects of game immersion (Sander Huiberts)• 1- Being transported into the game world• 2- Absorption in the activity• 3- Identification with the situation or a character of the game

– EMOTIONAL RESPONSE (GENERATING EMPATHY)– BRECVEMA model of emotion reduction

3

Definition: Serious Games

Serious games are games that have purposes other than entertainment, such as:

u Education (children learn how to read, count, play musical instruments)

u Advertisement (brand logos in the game: Coca Cola, Whiskas)

u Simulation gaming (firemen, rescue service)u Health (wrist rehabilitation)u Games with a purpose (science)

9

l Serious games are 10-15% of the market worldwidel But — around 40% in the Netherlands!

Serious games according to Game monitor

10

Job market

11

Expected development of serious games

13

Games with a Purpose (GWAP)

… have a purpose to:

l Collect scientific datal Outsource computation to humans that computers cannot do yet l Collect the ground truth data for machine learning algorithms

14

Games with a Purpose: principles

l Humans are much better at some tasks than computers (common-sense knowledge, perception, vision)

l These tasks can be fun to solve

15

16

ESP Game

17

ESP Game

Louis Van Ahn:

Google talk 2006

15 Million agreementswith 75.000 players

Many people play over 20 hours a week

18

l 50 million agreementsl 200,000 playersl A lot of fun and devotion!

ESP Game

19

Sample label

l Bushl George l Manl Presidentl Dumbl Yuck

Louis Van Ahn: Google talk 2006

20

Players about ESP game

l “It's so much fun to guess what others think. You have to step outside of yourself to match”

l “When your view on the world is the same it gives a sense of connection that lovers would envy”

l “Strangely addictive”l “Helps me learn English”

21

How about we apply ESP game to music?

22

l Luis von Ahn → PhD Student Edith Law

How about we apply ESP game to music?

23

How about we apply ESP game to music?

24

• Goal: enable music annotation with tags which are less subjective than those from social websites (e.g. last.fm)

• Method: randomly paired with a partner, both label short musical excerpts

• Question: Do you both listen to the same music? (after seeing each other’s tags)

• Dataset created: Magnatagatune• ~25000 29s long music clips, each of them annotated with a

combination of 188 tags

Tag a Tune

25

g Law & Van Ahn (2009), Input-Agreement: A New Mechanism for Collecting Data Using Human Computation Games

26

• Goal: enable music annotation with tags which are less subjective than those from social websites (e.g. last.fm)

• Method: randomly paired with a partner, both label short musical excerpts

• Question: Do you both listen to the same music? (after seeing each other’s tags)

• Used for evaluating algorithms that automatically tag music (ISMIR, 2009)

• human players can correctly guess that the music are the same 93% of the times when paired against the aggregate bot, while only approximately 70% of the times when paired against an algorithm bot.

Tag a Tune

28

Michael Mandel and Dan Ellis (ISMIR proceedings, 2007)

Our main goal, which shaped the design of the scoring rules, was toencourage users to describe the music thoroughly, to be original, yet relevant.

MajorMiner

29

MajorMiner

l You will be presented with one randomly selected 10-second music clip at a time.

l Describe that clip with a word or phrase (we call them "tags”...) and press enter.

l If you're the first person to describe that clip with that tag, you'll get 2 points when the next person tags that clip with that tag.

l If you're the second person to describe that clip with that tag, you'll get 1 point immediately.

l If more than two people have already tagged that clip with that tag, you won't get any points, but you can try another tag.

l Tag each clip as many times as you want, follow one of the "new clip" links to listen to a new one.

30

Spot the Odd Song Out

https://smcse.city.ac.uk/doc/mirg/casimir/game/camir_gameClient/index.php

31

In our work, the main goal is achieving a wise mass to assist in recruitment and measurement of musical talent … (Gomes et al., 2013)

32

MoodSwings

A music emotion labeling game• Create automated solutions for labeling the emotional

content of music

33

MoodSwings

Russel’s Valence/Arousal Model

Russel (1980): Valence and Arousal

34

MoodSwings

R. E. Thayer, The Biopsychology of Mood and Arousal. Oxford, U.K.: Oxford Univ. Press, 1989.

Advantages:• No hassle with tags• Continuous values• Can easily model changes over time

36

MoodSwings: Gameplay

1. The player’s cursor blinks every second.2. After 5 seconds, partner’s cursor becomes visible.3. The partner’s cursor blinks once every 3 seconds.4. The size of both cursors decreases.5. At the end of each round, the player’s score is presented.6. At the end of the match, the artist-title list is presented.

37

MoodSwings: Scoring

g Primary points:Ø Overlap between cursorsØ Accumulated when partner’s cursor is visible.

g Bonus points:Ø Convincing the partnerØ After staying stationary for at least 1 second, stationary player earns

5 points if partner moves towardsØ May be collected each second.

38

MoodSwings: Test phase

• Bias towards highvalence and arousal

• Avoidance ofextremes

39

MoodSwings: Data analysis

Within two years:g 5,177 gamesg Over 150,000 valence-arousal labelsg 1,158 songsg 16% coverage of all available songs

Gameplay scenarios:g Two playersg Player vs AIg Player vs Recorded

g Coldstart problem

40

MoodSwings: Data analysis

Most movements occur at the beginning

Analyzing Label vs Time

41

Game research in Utrecht...

42

Emotify: Student presentation

Challenge:Expressed vs. induced emotion in music

43

Hooked

A song starts playing, usually from somewhere in the middle.

The player has 15 s to decide whether they know the song.

44

Once the player guesses,themusic mutes for 3 s. While the sound is mutes, players sing along in their head. After 3 seconds, music starts playing again.

Hooked

45

Sometimes the music is what should be playing and sometimes it is a few seconds offset. Players who guess correctly win all the points at stake; players who fail lose the points.

Hooked

46

Hooked on Musichttp://www.hookedonmusic.org.uk

http://www.hookedonmusic.org.uk

47

The most recognizable song of all time ...

… according to the Hooked game?

48

Design questions for musical GWAPs

l Where to get properly licensed (Creative Commons licensed?) music

l How to validate that answers are seriousl How to find balance between solving serious tasks

and having fun

49

Garbage data prevention

l Give points for reaching consensusl Randomly add already correctly labeled itemsl Add verification step l “Psychological quiz”-style reward

50

Serious games for music education

51

Steve Reich’s Clapping music app

g Serious game: training of rhythm skillsg Steve Reich: “Clapping music (1972)”

https://www.youtube.com/watch?v=FcFyl8amoEE

g What is this? https://www.youtube.com/watch?v=lzkOFJMI5i8

g Steve Reich on the app: https://www.youtube.com/watch?v=7Z23EmPsoto

g Collaboration: LondonSinfonietta, Queen Mary University of London, Touchpress, started in 2015

52

Steve Reich’s Clapping music app

g Game with a purposeg Research at Queen Mary U of London: How do we learn

rhythm?• 1.How are musical performance skills acquired through a digital

game experience?

• 2.Can audience engagement with a new music genre be increased by a game based smartphone app?

• How can we measure rhythmic complexity of patterns?

53

g Rhythm complexity:

Steve Reich’s Clapping music app

1" 2" 1"2" 2" 1" 1" 2"

1" 2" 1"2" 2" 1" 1" 2"

Pa&ern"4"

Pa&ern"5"

2" 1"2" 2"1" 1"

Pa&ern"1"

2" 1"

54

g Ongoing research! Stay tuned! Play the game!

Steve Reich’s Clapping music app

1" 2" 1"2" 2" 1" 1" 2"

1" 2" 1"2" 2" 1" 1" 2"

Pa&ern"4"

Pa&ern"5"

2" 1"2" 2"1" 1"

Pa&ern"1"

2" 1"

Research: http://cogsci.eecs.qmul.ac.uk/clappingmusicresearch/Home.html

55

Rhythm games

g most prominent genre in music-related video games (Fleshner, 2016)

g players: actively respond to visual and auditory cues tomimic rhythmic patterns

g Dance, Dance Revolution; Guitar Herog DDR: improves sight reading skills, e.g. Chunking and reading

ahead (Auerbach et al, 2010)g DDR: https://www.youtube.com/watch?v=NqD-B8t_6us#t=100s

g DDR: foot-based controller, Guitar Hero: handheld controllerg Lots of games mimics for iPad using touch screen (e.g. Groove

coaster, Tone sphere)

56

Rhythm games involving pitch

g pedagogical and cognitive reasons for combining rhythmand pitch to increase engagement with andeducational usefulness of the game (Fleshner, 2016)

g “The interaction between rhythmic and metric features on the one hand and tonal function on the other is soimportant that it should be instilled continually from the earliest possible point” (Karpinksi, 2000)

g Circadia (Puzzle game): https://www.youtube.com/watch?v=3InmSV4FRQY

g Soundrop: https://www.youtube.com/watch?v=63FyqOMpORI

g Not reaction to stimulus, but creationg Both compositional tool and instrument

57

Conclusion

g Serious games for music exist in various facets

g GWAP: collecting data for research on sound and musicg Specifically for MIR

g Serious games specifically for music educationg What technolgical and musical knowledge is required for

designing these games?• Example Chordify http://chordify.net/chords/kensington-home-again-official-video-kensingtonband

g Creating new instruments, new forms of interaction• Ge Wang’s Ocarina from Smule: https://www.youtube.com/watch?v=ol1b5mMeP7s#t=609s

g Going to be increasingly important for health care