Download - Music and Machine Learning
![Page 1: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/1.jpg)
0.5setgray0
0.5setgray1
Music and Machine LearningUsing Machine Learning for the Classification of
Indian Music: Experiments and Prospects
Paritosh K. Pandya
School of Technology and Computer Science
Tata Institute of Fundamental Research
email: [email protected]
http://www.tcs.tifr.res.in/∼pandya
SNDT 2006 – p.
![Page 2: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/2.jpg)
Outline
Motivation (Example)
Introduction to Machine Learning
Intelligent Music Processing Examples
Indian Music: Some questions
Automatic Raag Recognition System
SNDT 2006 – p.
![Page 3: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/3.jpg)
Computers and Information Processing
Evolution of computers
Scientific calculations: e.g. planetary orbits
Data processing: e.g. inventory
Multimedia Rich Text, Graphics, Pictures, Animations,Video, Sound and Music.Computers can store, edit, process and display all ofthese!!
Internet and World-wide Web:
SNDT 2006 – p.
![Page 4: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/4.jpg)
The Classical Music Archives
SNDT 2006 – p.
![Page 5: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/5.jpg)
Computers and Arts
Computers and networks are increasingly used forstoring, processing (editing, cataloguing, searching),and disseminating artistic content.
Web Portals with artistic, educational andresearch-oriented content are becoming available e.g.complete works of Shakespeare
Computers can be used to analyse artistic content innew and sophisticated manners.
Computer as a tool for research in humanities and arts:Example Discovery channel news (2003): Computers toreveal Shakespeare
SNDT 2006 – p.
![Page 6: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/6.jpg)
Learning Machines
Traditionally, computers are calculating devices. How tocalculate must be fully pre-programmed.
People observe patterns in nature, they discover rulesand they learn.
Can Computers learn? A question addressed byartificial intelligence.
Learning System A system capable of the autonomousacquisition and integration of knowledge.
SNDT 2006 – p.
![Page 7: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/7.jpg)
How do systems learn?
Supervised Learning: Learn from examples.
Training Example Set (annotated)
Feature Selection (Input)
Target function representation
Statistical learning and classification
X
y
y = Number ofoccurrences of Ma after<Ga,MaTiv,Pa>
x = Number of oc-currences of Re after<Ga,MaTive,Pa>
SNDT 2006 – p.
![Page 8: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/8.jpg)
Neural Nets
Learning functions
Given monthly icecream sales and average temperature forlast 10 years, predict icecream sales this summer.
SNDT 2006 – p.
![Page 9: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/9.jpg)
Why use learning system?
The relationship between data elements is notformalized. Only examples are available.
Relationship between data items is buried within largeamount of data.
Data mining: using historical data to discover relationshipsand using this to improve future performance.
SNDT 2006 – p.
![Page 10: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/10.jpg)
Applications of Machine Learning
Speech recognition
Image recognition (Face recognition)
Identifying Genes
Predicting Drug Activity
Cataloguing Faint Objects in Astronomical Data
Detecting Credit Card Frauds
Predicting Medical Outcomes from Historic Data
Detecting Hacking and Intrusion from Network Load
Computational Linguistics
SNDT 2006 – p. 10
![Page 11: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/11.jpg)
Music Performance Visualisation
Performance worm [Widmer, Vienna]
Different players have different ways of building tensionor expression in the music
Measure subtle changes in beat level tempo versusloudness for each note played.
Represent this visually in tempo-loudness space as atrajectory called "performance worm".
SNDT 2006 – p. 11
![Page 12: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/12.jpg)
Our Experiments
Real-time melody tracker (Click)
Time (s)0 15
Pitc
h (H
z)
120
300
sa
sa
re
ga
ma
pa
dha
ni
SNDT 2006 – p. 12
![Page 13: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/13.jpg)
Recognition of Concert Pianists
Characterisation of Personal Expression FeaturesClassification
Classification between 22 piano players
Classification based on performance worm like data
Achieved accuracy comparable with human listeners.
[Saunders et al (2005)]
SNDT 2006 – p. 13
![Page 14: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/14.jpg)
Islands of Music
Intelligent structuring and exploration of digital musiccollections [Pampalk et al (2004)]
Grouping of Music bySimilarity
Genre and Style
Performer
Timbral and rythmic con-tent
Automatic classification of music by Genre: Classical,Country, Disco, HipHop, Jazz and Rock [Pye 2000]About 90% success on 176 songs
SNDT 2006 – p. 14
![Page 15: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/15.jpg)
Music Structure Analysis
Structure in Music Composition
repetition, transposition, call and response, rythmicpatterns and harmoic sequences
shape of a song e.g. AABA
Automatic structure analysis attempts to discover suchstructure [Danenberg,CMU]
Beat tracking and Tempo Detection
Identifying time signatures and tempo
Marking beat positions within music [Simon Dixon]
SNDT 2006 – p. 15
![Page 16: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/16.jpg)
Music representation: Audio
Audio Waveform Santur playing(Click)
Rich set of features: Pitch, Amplitude, Spectra
SNDT 2006 – p. 16
![Page 17: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/17.jpg)
Structured Music Representation
Example: Musical Score Notation
SNDT 2006 – p. 17
![Page 18: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/18.jpg)
Computers and Music Notation
MIDI files: computer representation of musical score.
Can be recorded from keyboards etc.
Synthesizers: MIDI → Sound
Issue Expressive Music Representation(a hot topic for research!)
Music Notation for Indian Music
Bhatkhande or Paluskar systems
Not used by professional musicians
Lacks structures
SNDT 2006 – p. 18
![Page 19: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/19.jpg)
Swarupa: Structured Music
define kaida2 as pat(
8.pat(dha,te,te,dha | te,te,dha,dha |
te,te,dha,ge | ti,na,ke,na),
8.pat(ta,te,te,ta | te,te,ta,ta |
te,te,dha,ge | dhin,na,ge,na) );
define palta1 as pat(
8.pat(dha,te,te,dha,te,te,dha,dha |
dha,te,te,dha,te,te,dha,dha),
8.pat(dha,te,te,dha,te,te,dha,dha |
te,te,dha,ge,dhin,na,ge,na),
8.pat(ta,te,te,ta,te,te,ta,ta |
ta,te,te,ta,te,te,ta,ta),
8.pat(dha,te,te,dha,te,te,dha,dha |
te,te,dha,ge,dhin,na,ge,na) );
SNDT 2006 – p. 19
![Page 20: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/20.jpg)
Swarupa: Structured Music
define kaida2n as[A::[ 2:dhatita::[dha,te,te], dhadha::[dha,dha]
tite::[te, te], dha,ge | tin,na,ke,na ];B::[ A.1{khali} | C::[A.2 | A.3{bhari}] ]];
define palta1 as[ 3:[A.1 |] C; 3:[B.1 |] C; ];
define palta2 as[ A.1 | D::4:5%[te,te] ; B ;
B.1 | D ; B ];
Synthesis Swarupa → Audio (See MuM Webpage)Music transcription: Audio → Score
SNDT 2006 – p. 20
![Page 21: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/21.jpg)
Indian Music Research using AI
Some topics
Classification of MusicRaag RecognitionClassification of Music in Thaats and JatisClassification of Raags into Time Cycle, SeasonalCycles,Classification of music by gharanasPerformer recognitionIdentification of Raag LakshansAssociation of Bhaav with musical performance
[B.Chaitanya Deva, 1981]
Beat tracking and taal recognition
Identification of musical structureSNDT 2006 – p. 21
![Page 22: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/22.jpg)
Associated Applications
Music Visualisation
Musical query processing from large annotated musicaldatabases.
Automatic music composition
Automatic accompaniment
Pursuit: Distance Education of Indian Music
SNDT 2006 – p. 22
![Page 23: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/23.jpg)
Machine Recognition of Raags
Raga performance as sequence of notes.Stop Sa Re Ga Pa Ga Re Sa Stop Dha Sa Re Ga
Sequential pattern classification problem
Data is not unordered set of samples.
Data elements occur in an order: spatial or temporal.
The probability of next data element crucially dependson the order of occurrence of preceeding elements.
Hidden Markov Models (HMM) are widely used.
SNDT 2006 – p. 23
![Page 24: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/24.jpg)
Finite state automaton for Raag
Bhoopali
2Ga
3Re
4Pa
5Sa
6Dha
7Stop System can be in one of
finite number of states
Current state depends onthe past and current inputseen
Current state and next in-put determines the possiblenext states
Experiments: Manual con-struction of raag automatabased on Bhatkhande Books[Sahasrabuddhe]
SNDT 2006 – p. 24
![Page 25: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/25.jpg)
Hidden Markov Model of a Raag
Bhoopali
2Ga 1.00
0.09
3Re 1.00
0.45
4Pa 1.00
0.235
Sa 0.99
0.06
0.51
0.41
0.45
0.076
Dha 1.00
0.41
0.38
0.07
0.32
0.62
0.30
7Stop 1.00
0.46
0.05
0.18
0.25
0.06
Probability ofseeing a note inthe given state.
Probability ofmoving from onestate to another
An HMM model canbe learnt from train-ing data
Analysis Given anHMM and a note se-quence, compute itsprobability of occur-rence.
SNDT 2006 – p. 25
![Page 26: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/26.jpg)
Raag Recognition using HMM
Hidden Markov Model for a raag
Finite state automata
Probability of “seeing a note” in each state.
Probability of transition between states.
HMM model can be learnt from a set of training data
Given a note seqeuence we can compute its probabilitywithin given Raag HMM model.
SNDT 2006 – p. 26
![Page 27: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/27.jpg)
Kansen: A raga recognition system
An Experiment at TIFR using a Toolkit HTK:
Learns HMM for each raag from training data(Baum-Welch Algorithm)
Training data: (Bhatkhande, IITK) collection of midi filesof raags played on keyboard. We use 29 raag database.
Test data: sequence of notes
Output: probability of the sequence being in each raag.
Preliminary Results
About 86 percent success on 29 raag recognition
Confusion between close raags
Insufficiency of dat a significant reason
(Joint work with Bhaumik Choksi and K. Samudravijaya)SNDT 2006 – p. 27
![Page 28: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/28.jpg)
Bhatkhande
MIDI File Database of Indian Raags (IIT, Kanpur)
Adana, AheerBhairav, AlhiyaBilawal, Bageshri,Bahar, Basant, BasantMukhari, Behag, Bhoopali,BhoopaliTodi, ChandraKauns, ChhayaNut, Des,Durga, Gaud, Hamir, JataShwari, JaunaPuri Jogiya,Lalit, Malkauns, MiyankiMalhar, Multani, Pahadi,Peelu Sohini, TilakKamode, Tilang, Todi
Each midi file created by playing from Keyboard (e.g.Des.mid)
Basic database of 29 Raags (above)
Full database of 300+ Raags
SNDT 2006 – p. 28
![Page 29: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/29.jpg)
Demonstration
Input Stop Sa Sa DhaKo Sa GaKo Re Stop Sa Ga Ga MaGaKo Re GaKo Re Sa Ni DhaKo Ni Sa Re Sa Ni SaOutput Log of probability of being in a raagI=1 t=0.02 W=ChandraKauns v=1
I=2 t=0.02 W=ChhayaNut v=1
I=3 t=0.02 W=Hamir v=1
I=4 t=0.02 W=Pahadi v=1
I=5 t=0.02 W=MiyankiMalhar v=1
I=6 t=0.02 W=Adana v=1
I=8 t=0.02 W=Peelu v=1
J=0 S=0 E=1 a=-164.11 l=0.000
J=1 S=0 E=2 a=-163.60 l=0.000
J=2 S=0 E=3 a=-160.92 l=0.000
J=3 S=0 E=4 a=-160.72 l=0.000
J=4 S=0 E=5 a=-158.36 l=0.000
J=5 S=0 E=6 a=-117.88 l=0.000
J=13 S=0 E=8 a=-88.55 l=0.000
Summary Peelu (-88) Adana (-117) MiyankiMalhar (-158)SNDT 2006 – p. 29
![Page 30: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/30.jpg)
Demonstration (cont)
Stop NiKo Dha Ni Sa NiKo Pa Ma Pa GaKo Ma Re SaBahar (-20.5) MiyankiMalhar (-23) Adana (-53)
Stop Ma Pa NiKo Dha Ni Ni Sa Stop Ni Sa Re GaKoGaKo Ma Re SaMiyankiMalhar (-52) Bahar (-66) Adana (-75)
Stop Ma Pa Ni Ni Sa Ni Sa Sa Stop Pa Ni Sa Re NiKoDha Pa Stop Pa Dha Ma Ga Re Stop Ga Re Ni SaDes (-127) Gaud (-139) MiyankiMalhar (-144)
Stop Ga Ma DhaKo DhaKo Pa Stop Ma Pa GaKo MaReKo Sa Stop Ga Ma Pa DhaKo Ni Sa DhaKo PaBasant Mukhari (-105) Peelu (-106) Jogiya (-130)
SNDT 2006 – p. 30
![Page 31: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/31.jpg)
Demonstration (cont)
Stop Ni Sa GaKo ReKo Sa Stop Ni Sa GaKo MaTiv PaStop GaKo MaTiv Pa Ni Sa DhaKo Pa MaTiv GaKoReKo SaMultani (-76) Todi (-105) ChandraKauns (-169)
Stop DhaKo Ni Sa ReKo GaKo Stop ReKo GaKo ReKoSa Stop Sa ReKo GaKo MaTiv ReKo GaKo ReKo SaTodi (-58) Bhoopali Todi (-83) Multani (-124)
SNDT 2006 – p. 31
![Page 32: Music and Machine Learning](https://reader036.vdocuments.site/reader036/viewer/2022081400/54c6e5904a795974608b45c5/html5/thumbnails/32.jpg)
Conclusions
Computer analysis and machine learning providesinteresting new method of analysing music. It allowsmany intuitive and qualitative observations to be madeobjective, precise and quantitative.
Research with computational techniques lead to directapplications in music technology.
Intelligent music analysis is almost untried for IndianMusic.
Work requires collaboration between musicologists,computer scientists and electrical engineers.
Music researchers must help by building corpuses andannotated datasets for future machine analysis.
SNDT 2006 – p. 32