2
Recap
g Sound and music for interactivity and immersion in games.• How to achieve interactive music? (Michael Sweet):
• Method 1: Vertical Remixing (Layering)• Method 2: Horizontal Re-sequencing
• Immersion: 3 different aspects of game immersion (Sander Huiberts)• 1- Being transported into the game world• 2- Absorption in the activity• 3- Identification with the situation or a character of the game
– EMOTIONAL RESPONSE (GENERATING EMPATHY)– BRECVEMA model of emotion reduction
3
Definition: Serious Games
Serious games are games that have purposes other than entertainment, such as:
u Education (children learn how to read, count, play musical instruments)
u Advertisement (brand logos in the game: Coca Cola, Whiskas)
u Simulation gaming (firemen, rescue service)u Health (wrist rehabilitation)u Games with a purpose (science)
9
l Serious games are 10-15% of the market worldwidel But — around 40% in the Netherlands!
Serious games according to Game monitor
13
Games with a Purpose (GWAP)
… have a purpose to:
l Collect scientific datal Outsource computation to humans that computers cannot do yet l Collect the ground truth data for machine learning algorithms
14
Games with a Purpose: principles
l Humans are much better at some tasks than computers (common-sense knowledge, perception, vision)
l These tasks can be fun to solve
17
ESP Game
Louis Van Ahn:
Google talk 2006
15 Million agreementswith 75.000 players
Many people play over 20 hours a week
20
Players about ESP game
l “It's so much fun to guess what others think. You have to step outside of yourself to match”
l “When your view on the world is the same it gives a sense of connection that lovers would envy”
l “Strangely addictive”l “Helps me learn English”
24
• Goal: enable music annotation with tags which are less subjective than those from social websites (e.g. last.fm)
• Method: randomly paired with a partner, both label short musical excerpts
• Question: Do you both listen to the same music? (after seeing each other’s tags)
• Dataset created: Magnatagatune• ~25000 29s long music clips, each of them annotated with a
combination of 188 tags
Tag a Tune
25
g Law & Van Ahn (2009), Input-Agreement: A New Mechanism for Collecting Data Using Human Computation Games
26
• Goal: enable music annotation with tags which are less subjective than those from social websites (e.g. last.fm)
• Method: randomly paired with a partner, both label short musical excerpts
• Question: Do you both listen to the same music? (after seeing each other’s tags)
• Used for evaluating algorithms that automatically tag music (ISMIR, 2009)
• human players can correctly guess that the music are the same 93% of the times when paired against the aggregate bot, while only approximately 70% of the times when paired against an algorithm bot.
Tag a Tune
28
Michael Mandel and Dan Ellis (ISMIR proceedings, 2007)
Our main goal, which shaped the design of the scoring rules, was toencourage users to describe the music thoroughly, to be original, yet relevant.
MajorMiner
29
MajorMiner
l You will be presented with one randomly selected 10-second music clip at a time.
l Describe that clip with a word or phrase (we call them "tags”...) and press enter.
l If you're the first person to describe that clip with that tag, you'll get 2 points when the next person tags that clip with that tag.
l If you're the second person to describe that clip with that tag, you'll get 1 point immediately.
l If more than two people have already tagged that clip with that tag, you won't get any points, but you can try another tag.
l Tag each clip as many times as you want, follow one of the "new clip" links to listen to a new one.
31
In our work, the main goal is achieving a wise mass to assist in recruitment and measurement of musical talent … (Gomes et al., 2013)
32
MoodSwings
A music emotion labeling game• Create automated solutions for labeling the emotional
content of music
34
MoodSwings
R. E. Thayer, The Biopsychology of Mood and Arousal. Oxford, U.K.: Oxford Univ. Press, 1989.
Advantages:• No hassle with tags• Continuous values• Can easily model changes over time
36
MoodSwings: Gameplay
1. The player’s cursor blinks every second.2. After 5 seconds, partner’s cursor becomes visible.3. The partner’s cursor blinks once every 3 seconds.4. The size of both cursors decreases.5. At the end of each round, the player’s score is presented.6. At the end of the match, the artist-title list is presented.
37
MoodSwings: Scoring
g Primary points:Ø Overlap between cursorsØ Accumulated when partner’s cursor is visible.
g Bonus points:Ø Convincing the partnerØ After staying stationary for at least 1 second, stationary player earns
5 points if partner moves towardsØ May be collected each second.
39
MoodSwings: Data analysis
Within two years:g 5,177 gamesg Over 150,000 valence-arousal labelsg 1,158 songsg 16% coverage of all available songs
Gameplay scenarios:g Two playersg Player vs AIg Player vs Recorded
g Coldstart problem
43
Hooked
A song starts playing, usually from somewhere in the middle.
The player has 15 s to decide whether they know the song.
44
Once the player guesses,themusic mutes for 3 s. While the sound is mutes, players sing along in their head. After 3 seconds, music starts playing again.
Hooked
45
Sometimes the music is what should be playing and sometimes it is a few seconds offset. Players who guess correctly win all the points at stake; players who fail lose the points.
Hooked
48
Design questions for musical GWAPs
l Where to get properly licensed (Creative Commons licensed?) music
l How to validate that answers are seriousl How to find balance between solving serious tasks
and having fun
49
Garbage data prevention
l Give points for reaching consensusl Randomly add already correctly labeled itemsl Add verification step l “Psychological quiz”-style reward
51
Steve Reich’s Clapping music app
g Serious game: training of rhythm skillsg Steve Reich: “Clapping music (1972)”
https://www.youtube.com/watch?v=FcFyl8amoEE
g What is this? https://www.youtube.com/watch?v=lzkOFJMI5i8
g Steve Reich on the app: https://www.youtube.com/watch?v=7Z23EmPsoto
g Collaboration: LondonSinfonietta, Queen Mary University of London, Touchpress, started in 2015
52
Steve Reich’s Clapping music app
g Game with a purposeg Research at Queen Mary U of London: How do we learn
rhythm?• 1.How are musical performance skills acquired through a digital
game experience?
• 2.Can audience engagement with a new music genre be increased by a game based smartphone app?
• How can we measure rhythmic complexity of patterns?
53
g Rhythm complexity:
Steve Reich’s Clapping music app
1" 2" 1"2" 2" 1" 1" 2"
1" 2" 1"2" 2" 1" 1" 2"
Pa&ern"4"
Pa&ern"5"
2" 1"2" 2"1" 1"
Pa&ern"1"
2" 1"
54
g Ongoing research! Stay tuned! Play the game!
Steve Reich’s Clapping music app
1" 2" 1"2" 2" 1" 1" 2"
1" 2" 1"2" 2" 1" 1" 2"
Pa&ern"4"
Pa&ern"5"
2" 1"2" 2"1" 1"
Pa&ern"1"
2" 1"
Research: http://cogsci.eecs.qmul.ac.uk/clappingmusicresearch/Home.html
55
Rhythm games
g most prominent genre in music-related video games (Fleshner, 2016)
g players: actively respond to visual and auditory cues tomimic rhythmic patterns
g Dance, Dance Revolution; Guitar Herog DDR: improves sight reading skills, e.g. Chunking and reading
ahead (Auerbach et al, 2010)g DDR: https://www.youtube.com/watch?v=NqD-B8t_6us#t=100s
g DDR: foot-based controller, Guitar Hero: handheld controllerg Lots of games mimics for iPad using touch screen (e.g. Groove
coaster, Tone sphere)
56
Rhythm games involving pitch
g pedagogical and cognitive reasons for combining rhythmand pitch to increase engagement with andeducational usefulness of the game (Fleshner, 2016)
g “The interaction between rhythmic and metric features on the one hand and tonal function on the other is soimportant that it should be instilled continually from the earliest possible point” (Karpinksi, 2000)
g Circadia (Puzzle game): https://www.youtube.com/watch?v=3InmSV4FRQY
g Soundrop: https://www.youtube.com/watch?v=63FyqOMpORI
g Not reaction to stimulus, but creationg Both compositional tool and instrument
57
Conclusion
g Serious games for music exist in various facets
g GWAP: collecting data for research on sound and musicg Specifically for MIR
g Serious games specifically for music educationg What technolgical and musical knowledge is required for
designing these games?• Example Chordify http://chordify.net/chords/kensington-home-again-official-video-kensingtonband
g Creating new instruments, new forms of interaction• Ge Wang’s Ocarina from Smule: https://www.youtube.com/watch?v=ol1b5mMeP7s#t=609s
g Going to be increasingly important for health care