11/1/11  

1  

CompMusic: Computational models for the discovery of the world’s music

Xavier Serra Music Technology Group

Universitat Pompeu Fabra, Barcelona (Spain)

ERC mission: support investigator-driven frontier research.

CompMusic is funded with an ERC Advanced Grant for a period of 5 years and with a budget of 2,5 million Euros.

Current IT problems

•  IT research does not respond to the world's multi-cultural reality.

•  Data models, cognition models, user models, interaction models, ontologies, … are culturally biased.

•  Music information is not just CDs and metadata.

Taxonomy of musical information

(Lesaffre, 2005)

11/1/11  

2  

Computational music modeling

Sound and Music Computing

Music Information Processing

Computational Musicology

Hum

an-Computer

Interaction

Ontologies

Cognitive Musicology

Cognition models

Interaction models

Data models

CompMusic objectives

•  Promote a multicultural approach to IT research.

•  Advance in the description and formalization of music to make it accessible to computational approaches.

•  Reduce the gap between audio signal descriptions and semantically meaningful concepts for music.

•  Develop data modelling techniques for different music repertories.

•  Develop computational models to represent culture specific musical contexts.

•  Design culture driven music discovery systems.

Proposed approach •  Combination of academic disciplines: Computational

Musicology, Cognitive Musicology, Music Information Processing, Music Interaction.

•  Combination of methodologies: qualitative and quantitative; scientific and engineering.

•  Combination of information sources: audio features, symbolic scores, text commentaries, user evaluations, etc…

•  Combination of music repertoires: Indian (hindustani, carnatic), Turkish-Arab (turkish, andalusian), Chinese (han).

•  Combination of cultural perspectives: Research teams and users immersed in the different music cultures.

Why these musical repertoires?

•  Belong to formalized classical traditions with strong influence on current society.

•  Musicological and cultural studies available.

•  Alive performance practice traditions.

•  Exists within active social/cultural contexts.

Possibility to challenge current western centred information paradigms.

11/1/11  

3  

Indian music Melodic structure: Raga

Rhythmic structure: Tala

Texture: monophonic

Style: pre-composed and improvisatory.

…

Hindustani: Ravi Shankar

Carnatic: Sudha Ragunathan

Turkish-Arab music Melodic structure: Maqam

Rhythmic structure: Wazn

Texture: Monophonic

Style: pre-composed and improvisatory.

Ottoman classical music

Andalusian classical music

Han Chinese music Melodic structure: heptatonic (not pentatonic!!)

Harmony: five harmonies

Rhythmic structure: duple

Texture: Polyphonic

Liu Ji Hong, Erhu concerto

CompMusic tasks

11/1/11  

4  

Task 1: Music repertoires

Gathering and organizing audio recordings, metadata, descriptions, scores, plus all the needed contextual information.

References: •  MTG-DB framework (MTG-UPF)

•  http://www.raaga.com; http://chinasite.com; http://www.listenarabic.com; http://www.turkishmusicportal.org; …

•  Open data movement: Wikipedia, Musicbrainz, Wikibooks, Wordnet, DBLP Bibliography, DBTune, Geonames, …

•  Resource Description Framework (http://www.w3.org/RDF/)

•  Grid Computing

Task 2: Musicological framework

Musicological studies to understand the chosen repertories within their cultural context.

References: •  Tonal pitch space theory (Lerdahl, 2001)

•  Performance studies (Gabrielsson, 2003)

•  Embodied cognition (Leman, 2008)

•  Humdrum toolkit (http://humdrum.ccarh.org)

•  Rasas in Indian art (Rangacharya, 2010)

Task 3: Music ontologies

Building the ontologies needed for annotating the gathered collections.

References: •  The music ontology specification (http://musicontology.com)

(Raimond, 2007)

•  Community-based ontologies (Mika, 2006)

•  Knowledge management and metadata (Pachet, 2005)

•  http://musicbrainz.org; http://wordnet.princeton.edu

Task 4: Audio description

Audio content analysis to describe the music collections chosen.

References: •  Essentia & Gaia framework (MTG-UPF)

•  Music transcription (Klapuri & Davy, 2006)

•  Top-down and knowledge-based processing (in Klapuri & Davy, 2006)

•  Computational auditory scene analysis (Wang & Brown, 2006)

11/1/11  

5  

Task 5: User profiling

Characterization of users and communities, modelling their musical preferences and behaviours.

References: •  Social Computing (Chai et al., 2010)

•  Theory of music preferences (Rentfrow & Gosling, 2003)

•  http://www.last.fm; http://freesound.org

Task 6: Music interaction

Interaction models by studying user behaviour in musical tasks.

References: •  Cultural Computing (Nakatsu et al., 2010)

•  Information Foraging Theory (Pirolli, 2007)

•  Interactive Information Retrieval (Cole et al. 2005)

•  Table-top interfaces (Reactable)

Task 7: Music discovery

Active models and systems for culture-based music discovery.

References: •  Collaborative creativity

•  Online learning (Moh et al., 2008)

•  Recommendation systems (Celma, 2009)

•  http://freesound.org; http://last.fm

Conclusions

Big and challenging !!!!

But hopefully we can contribute with our music research to develop better

IT for our multicultural world.

11/1/11  

6  

References (1 of 3) •  Celma, O. 2008. Music Recommendation and Discovery in the Long

Tail. PhD thesis.

•  Cole, C., et al. 2005. "Interactive information retrieval: Bringing the user to a selection state". in A. Spink & C. Cole (Eds.), New directions in cognitive information retrieval. Springer.

•  Chai, S., et al. (Eds.) 2010. Advances in Social Computing. Springer.

•  Gabrielsson, A. 2003. “Music Performance Research at the Millennium”. Psychology of Music.

•  Klapuri, A., Davy, M. (Eds.) 2006. Signal Processing Methods for Music Transcription. Springer.

•  Leman, M. 2007. Embodied music cognition and mediation technology. The MIT Press.

•  Lerdahl, F. 2001. Tonal Pitch Space. Oxford University

References (2 of 3) •  Lesaffre, M. 2005. Music Information Retrieval: Conceptual framework,

Annotation and User Behaviour. PhD Thesis.

•  Mika, P. 2006. “Ontologies are us: A unified model of social networks and semantics”. Web Semantics: Science, Services and Agents on the World Wide Web 5, no. 1 (March): 5-15.

•  Moh, Y., Orbanz, P., Buhmann, J. M. 2008. "Music Preference Learning with Partial Information". ICASSP 2008.

•  Nakatsu, R. et al. (Eds.) 2010. Cultural Computing. Springer.

•  Pachet, F. 2005. “Knowledge Management and Musical Metadata”. Encyclopedia of Knowledge Management.

•  Pirolli, P. 2007. Information Foraging Theory: Adaptive Interaction with Information. Oxford, Oxford University Press.

References (3 of 3) •  Raimond, Y. 2008. A Distributed Music Information System. PhD

Thesis.

•  Rangacharya, A. 2010. The Natyasastra. Munshiram Manoharlal Publishers.

•  Rentfrow, P. J., Gosling, S. D. 2003. “The Do Re Mi's of Everyday Life: The Structure and Personality Correlates of Music Preferences”. Journal of Personality and Social Psychology 84, no. 6: 1236 -1256.

•  Wang, D. L. and Brown, G. J. (Eds.). 2006. Computational auditory scene analysis: Principles, algorithms and applications. IEEE Press/Wiley-Interscience.