semi-automated extraction of expressive performance...
TRANSCRIPT
![Page 1: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/1.jpg)
Semi-automated extraction of expressive performance information from acoustic recordings of piano music
Andrew Earis
![Page 2: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/2.jpg)
Outline
Parameters of expressive piano performanceScientific techniques: Fourier transform & wavelet transformCHARM: The semi-automated system
![Page 3: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/3.jpg)
Measurable features of expressive performance
• Timing (note onset and offset)• Dynamics (variations in intensity of notes and chords)• Pitch
time
amplitude
![Page 4: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/4.jpg)
Timing parameterisationNote onset time. The time at which the note is played.Note offset time. The time at which the note is released.Attack duration. The time from the initial sound until its amplitude is at a maximum.Sustain duration. The time from maximum sound amplitude until the note is released.Decay time. The time from when the note is released until the sound ceases.
![Page 5: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/5.jpg)
Dynamics parameterisationPower of noteMaximum amplitude (amplitude of attack)Sustain amplitude (amplitude, or change of amplitude, of the sustain)
![Page 6: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/6.jpg)
Pitch parameterisationFundamental frequency of noteFrequency of individual partials (and how they deviate from the precise harmonic frequencies)Vibrato
Frequency of vibratoAmplitude of vibrato
Glissando
![Page 7: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/7.jpg)
Aims
to extract the expressive performance information from acoustic recordings of piano music with reference to a digitised version of the musical score of the work being performed.
![Page 8: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/8.jpg)
Simple flowchart
Digitised musical score
Acoustic recording
Note-on-note synchronisation
algorithm
Event(note or chord)
onset timesdynamics
Individual noteonset timesoffset timesdynamics
Expressive performance
data of individual
performance
![Page 9: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/9.jpg)
MethodsThe location in time of individual notes andchords within an acoustic recording mustbe identified and related to a digitisedscore of the work being performed.
This is a classical time-frequency analysisproblem
![Page 10: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/10.jpg)
Fundamental research issuesA number of constraints on the time-frequency resolution required.
Required frequency resolutiondetermined by the interval between adjacent semitonesproportional to the frequency of the lower semitoneSpacing varies from about 1.6Hz at the bottom of the piano keyboard to over 120Hz at the topthe fundamental frequencies ranging from 27.5Hz to just over 4kHz.
Required timing resolutionThe shortest note duration in rapid note passages is around 50ms.The human ear is capable of perceiving differences of as little as 10ms.
![Page 11: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/11.jpg)
Possible approaches
Filterbank techniquesTime-frequency analysis (using Fourier Transforms, wavelets and spectrographs)Modelling
![Page 12: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/12.jpg)
Short Time Fourier analysis
![Page 13: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/13.jpg)
Wavelet analysis
![Page 14: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/14.jpg)
Problems with ‘simple approach’Synchronization between score and acoustic recording relies on stepping through each event in the score in turnIf a particular note or chord in the score is not played, the whole system can break downOther problems
too much noise in the recordinga high degree of rubatochords are played with too much asynchrony
The longer the passage of music to be analysed, the more likely it is that such a system will break down. Time-consuming manual checking of the data would also required to ensure accuracy.
![Page 15: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/15.jpg)
CHARM: Semi-automated system
A semi-automated expression extractor, requiring user input for checking and verifying results as the algorithm proceeds i.e. note-on-note.Input to system: MIDI score, acoustic waveformTechnology incorporated: filterbank/wavelet-based note onset detectorNote/event parameter measurements: onset time, dynamics
![Page 16: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/16.jpg)
Algorithm planSTEP 1
Digitisation of musical score
STEP 2Manual beat tapping to
acoustic recording
STEP 3Approximate
performance score→ Estimate of main beat onset
times
STEP 4Measurement of actual main beat onset times using
algorithm
STEP 5Manual correction of main beat onset
times
STEP 6Calculation of other event onset times and all event dynamics, individual note onset times /
dynamicsnote onset timings and event/individual note dynamics
![Page 17: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/17.jpg)
AdvantagesQuick to use, results can be verified as they go along – know they are correct!Very easy to useWide range of scores can be usedQuick and easy to develop and automate furtherResults are verified by the user as measurements are made, ensuring that all results are correct and can be used reliably for musicological analysisFurther enhancements can be made (e.g. increase level of automation, sophistication of parameters)
![Page 18: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/18.jpg)
STEP 1
Digitisation of musical scoreScore is scanned using a standard flatbed scanner Converted to symbolic data with SharpeyeErrors are corrected manuallySymbolic data is then converted to the Humdrum data formatThis is outputted as ASCII tabular data
![Page 19: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/19.jpg)
STEP 2
Manual beat tapping to acoustic recording
Standard beat tapping proceduresbackbeat.exeaverage of 20 complete taps initially!
![Page 20: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/20.jpg)
STEP 3Averaged manual beat tapped data combined with the digitized musical score
-> Approximate performance score
measured from beginning of workabsolute beat6
bar number5
1 = downbeat0 = beat-1 = offbeat
metric level4
MIDI number of note pitchnote3
Expected duration of note in ms based on score durationduration2
Average absolute time in ms of beat tapped timingsabstime1
DescriptionColumn
![Page 21: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/21.jpg)
Approximate performance score: Mazurka in A minor op 7 no 2
1912 646 76 1 0 02558 463 77 0 1 13021 154 76 -1 1 1.753175 603 57 0 1 23175 603 62 0 1 23175 603 65 0 1 23175 603 74 0 1 23778 652 57 1 1 33778 652 62 1 1 33778 652 65 1 1 33778 652 77 1 1 34430 1111 77 0 2 44914 627 60 0 2 54914 627 65 0 2 5
![Page 22: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/22.jpg)
STEP 4
Correction of ‘main beat’ onset times (from beat tapped times) using wavelet-based algorithm, programmed in MATLAB
![Page 23: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/23.jpg)
Parameters
Search window parametersWavelet parameter - kWavelet parameter - LPFWavelet tuningFeedback analysis
![Page 24: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/24.jpg)
Search window parameters
![Page 25: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/25.jpg)
Wavelet parameters - k
![Page 26: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/26.jpg)
Wavelet tuning
![Page 27: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/27.jpg)
Feedback analysis
![Page 28: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/28.jpg)
STEP 5
Manual correction of main beat onset times
Sonic Visualiser
Output – table of corrected main beat onset times
![Page 29: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/29.jpg)
STEP 6Estimation of subbeat onset times from main beat onset timesCorrection of estimated subbeat onset times using algorithmEstimation of main beat and subbeatdynamicsEstimation of individual note onset times
![Page 30: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/30.jpg)
End productMATLAB code converted to standard .exe filesInput data required
Excel control spreadsheet - for batch processing of data
Input/output file namesAnalysis parameters
‘Main beat’ beat tapped input data spreadsheet.wav sound recording
![Page 31: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/31.jpg)
End productThree different algorithms
mainbeatcorrection.execorrection of ‘main beat’ tapped onset timesmeasurement of main beat dynamics (in dB)
subbeatmeasurement.exemeasurement of sub beat onset timesmeasurement of sub beat dynamics
individualnotemeasurement.exemeasurement of individual note onset timings and dynamics
Will be available (or a variation of this) on CHARM website as downloadable resource
![Page 32: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/32.jpg)
End productFor example...
3 recordings of Chopin’s Mazurka in A minor op 7 no 2
Friedman (1930)Rubinstein (1939)Indjic (1988)
Graphs ofinter-onset interval for each event in scoredynamics of each event (scaled in dBs)
![Page 33: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/33.jpg)
Inter-onset interval
0
0.5
1
1.5
2
2.5
3
0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300
Beat
Friedman (1930)Rubinstein (1939)Indjic (1988)
![Page 34: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/34.jpg)
Event dynamics
-10.00
0.00
10.00
20.00
30.00
40.00
50.00
0 9 17 26 34 43 51 60 69 77 86 94 103
Beat
Friedman (1930)Rubinstein (1939)Indjic (1988)
![Page 35: Semi-automated extraction of expressive performance ...charm.cch.kcl.ac.uk/redist/pdf/s4Earis.pdf · Semi-automated extraction of expressive performance information from acoustic](https://reader030.vdocuments.site/reader030/viewer/2022041216/5e051a1bde9b12412676ff9b/html5/thumbnails/35.jpg)
Conclusions and future work
Further scientific development (wavelet parameters, shape, dynamics measurement)Extension to note offset timesTesting and refinement of user interface