large-scale capture of producer-defined musical semantics - ryan stables (semantic media @ the...
DESCRIPTION
This talk was given by Ryan Stables (Birmingham City University) at the "Semantic Media @ The British Library" event on 23 September 2013.TRANSCRIPT
Large-Scale Capture of Producer-Defined MusicalSemantics
Ryan StablesSchool of Digital Media Technology
Birmingham City University
Problem...
Problem Definition
Producer:
I Audio effects parametersusually refer to low-levelattributes.
I Professionally produced audiooften requires extensivetraining.
Researcher:
I Lack of semantically annotatedmusic production datasets.
I How can we map low-leveldescriptors to perceivedmuscial timbre?
Problem Definition
I Descriptors need to representthe views of music producers.
I These may change with genre,musical instruments, etc...
I Various terms may be used todefine similar things (colour,texture etc...)
Project Aims
1. Gather large amounts of semantics data during the musiccreation/production process.
I Develop a series of DAW plug-ins.I Extract information and anonymously upload it to a server.
2. Identify correlation and patterns in the semantics data.
3. Use the data to improve/aid music production tasks.
Model...
Project overview
Server
Descriptor name...
Save...Load...
Save...
Semantic Descriptor
Parameter Space
Feature Set
Pre/Post Gain
Analysis...
Natural Language
Processing
Dimensionality
Reduction
Etc...
(1)
(2)(3)
Figure : Schematic Overview of the Semantic Audio Feature Extraction Project.
(1) Plug-in interface
I Parameters can be setexperimentally.
I Semantic descriptors to bestored in text field.
I Descriptors can be loadedthrough same interface.
I Parameters are stored and/orset.
Figure : Semantic Audio plug-in: Multi-band distortion
(2) Feature Extraction
I Features are extracted from theselected region.
I The parameter space is stored.
I Semantic descriptors are sentas targets.
I Additional metadata is sent, ifavailable.
Server
Descriptor name...
Save...Load...
Save...
Semantic Descriptor
Parameter Space
Feature Set Pre/Post Gain
Analysis...
Natural Language Processing
Dimensionality Reduction
Etc...
Figure : Stored attributes.
(3) Mapping
I NLP Algorithms to identifysemantic correlation.
I Dimensionality reduction tofind correlation infeatures/parameters.
I Additional data partitionsbased on metadata (Genre,instrument, etc...)
I Results sent back to userplug-in.
Server
Descriptor name...
Save...Load...
Save...
Semantic Descriptor
Parameter Space
Feature Set Pre/Post Gain
Analysis...
Natural Language Processing
Dimensionality Reduction
Etc...
Figure : Results processing
Design Constraints...
Architecture
I Requirements:I Maximisation of user-base.I Transparency: Access to the processing chain.
I Design decisions:I Stand-alone plug-ins.I MultiFX.I Plug-ins within a plug-in.I Analysis-only.
I Other:I Free field vs. fixed word.I Before and after.I Metadata pane.
Analysis framework
I LibXtract.
I Hard-coded, C library.I Around 400 combined
audio features*.I [Bullock, 2007]
I Vamp.
I Plug-in within a plug-in.I Hosts LibXtract features,
amongst others.I [Cannam et al., 2006]
Mini-Project...
Mini-Project: Aims
I Analyse the production requirements of musicians.I Birmingham ConservatoireI The Music Producers GuildI The Birmingham Music Network
I Build a series of prototype systems for the collection ofmusical semantics data.
I Use these systems to collect data from a small group ofmusicians during the production process.
I Evaluate the results in order to identify a suitable system forfuture research.
I Demonstrate the feasibility of a wider research project in thisarea.
Mini Project: Schematic
Plug-in
development
Interface design
Algorithm
Development
Server, network,
data distribution
User Testing Data
Aquisition
Results
Analysis
Figure : Schematic Overview of the Mini-Project.
Positions and Timescale
I 2 x PhD Students: 1 x C4DM (QMUL) & 1 x DMT (BCU).
I 3 x Advisory roles.
I Timescale: 6-months from September 2013.
I Future: collaborative grant application.
References
Bullock, J. (2007).Libxtract: A lightweight library for audio feature extraction.In Proceedings of the International Computer MusicConference, volume 43.
Cannam, C., Landone, C., Sandler, M. B., and Bello, J. P.(2006).The sonic visualiser: A visualisation platform for semanticdescriptors from musical signals.In ISMIR, pages 324–327.