Audio-visual vibraphone transcription in real timeTiago F. Tavares #1, Gabrielle Odowichuck ∗2, Sonmaz Zehtabi #3, George Tzanetakis #4

# Department of Computer Science, University of VictoriaVictoria, Canada

1 tiagoft@uvic.ca, 3 szhetabi@uvic.ca, 4 gtzan@cs.uvic.ca

∗ Department of Electrical and Computer Engineering, University of VictoriaVictoria, Canada

2 godowichuk@gmail.com

Abstract—Music transcription refers to the process of detectingmusical events (typically consisting of notes, starting times anddurations) from an audio signal. Most existing work in automaticmusic transcription has focused on offline processing. In thiswork we describe our efforts in building a system for real timemusic transcription for the vibraphone. We describe experimentswith three audio-based methods for music transcription that arerepresentative of the state of the art. One method is basedon multiple pitch estimation and the other two methods arebased on factorization of the audio spectrogram. In additionwe show how information from a video camera can be used toimpose constraints on the symbol search space based on thegestures of the performer. Experimental results with varioussystem configurations show that this multi-modal approach leadsto a significant reduction of false positives and increases theoverall accuracy. This improvement is observed for all three audiomethods, and indicates that visual information is complimentaryto the audio information in this context.

Index Terms—Audiovisual, Music, Transcription

I. INTRODUCTION

There is an increasing trend of interfacing musical instru-ments with computers. The most common approach is to createspecialized digital instruments that send digital signals aboutwhat is played using protocols such as the Musical InstrumentDigital Interface (MIDI) or Open Sound Control (OSC) pro-tocols. However, these instruments frequently lack the hapticresponse and expressive capabilities of acoustic instruments. Amore recent approach has been to retrofit acoustic instrumentswith digital sensors to capture what is being played. Theseso called “hyperinstruments” [1] combine the haptic feel andexpressive capabilities of acoustic instruments while providingdigital control information. However, they tend to be expen-sive, hard to replicate, and require invasive modifications to theinstrument which many musicians dislike. Indirect acquisition[2] refers to the process of acquiring the control informationprovided by invasive sensors by analyzing the audio signalacquired by a microphone. When successful such systemscan provide similar capabilities to hyperinstruments withoutrequiring invasive modification. They are also easy to replicate,and have significantly lower cost.

MMSP’10, October 4-6, 2010, Saint-Malo, France.???-?-????-????-?/10/$??.?? c©2010 IEEE.

The focus of this paper is the indirect acquisition of controlinformation for the vibraphone which is a member of thepitched percussion family. The major components of a vibra-phone are aluminum bars, tube resonators placed under thebars, and a damper mechanism that controls whether notes aresustained after they are struck or not. There are some digitalcontrollers with similar layout to a vibraphone that simplytrigger digital samples and do not produce acoustic sound. Itis also possible to augment acoustic vibraphones with digitalsensors to obtain information about what bar is being playedand when. However, this is costly and hard to accomplishwithout altering the acoustic response of the instrument. In thispaper we describe how information about what bar is beingplayed and when can be extracted by computer analysis ofaudio-visual information. The audio signal acquired by a singlemicrophone, that is not attached to the instrument, is used asinput to automatic transcription algorithms. Such algorithmsrely on digital signal processing techniques to detect andcharacterize musical events (typically, the onset, offset andpitch of musical notes). The detection of musical notes isgenerally based on the harmonic model, that is, a note with acertain pitch f0 is modeled by a sum of M sinusoidal signalswhose frequencies are multiples of f0 and whose amplitudesand phases are individually defined [3], as in:

M∑m=1

am cos(2πmf0 + φm). (1)

The output of an automatic transcription algorithm is a setof notes characterized by their pitch (that is, which note inthe equal tempered scale is being played), onset (the instantwhen the note is played) and offset (the instant when thenote is damped). Figure 1(a) shows a piano roll representationof a short excerpt. In this representation, boxes representthe detected notes. They are positioned in the row of theircorresponding note and they span horizontally from their onsetto their onset times. The spectrogram corresponding that sameexcerpt is shown in Figure 1(b). The goal of our system isto produce the piano roll representation given as input thespectrogram as well as visual information from a camera. Thisprocess needs to be carried out in real time which requires allthe processing to be causal.

A4

G#4

G4

F#4

F4

E4

D#4

D4

C#4

C4

B3

A#3

0 0.5 1.0 1.5 2.0 2.5 3.0t (s)

(a) Piano roll.

0.0 0.5 1.0 1.5 2.0 2.5 3.0Time (s)

500

1000

1500

2000

2500

3000

Frequency

(H

z)

(b) Spectrogram.

Fig. 1. Piano roll representation and spectrogram for a short excerpt.

When several harmonic signals are summed, the joint detec-tion of their pitches becomes more difficult. This is especiallytrue in cases where the harmonic series of multiple signalsoverlap, as can be seen in Figure 1(b) where the third harmonicof note F4 coincides with the fourth harmonic of note C4 at0.55 seconds. This is reflected in the results of most automaticevent detection algorithms, which often yield results thatcorrespond to a harmonic or sub-harmonic of the ground truth.As music signals frequently contain combinations of noteswith overlapping harmonics this makes the separation anddetection of multiple sound sources in music more challenging,in some ways, than speech signals. Another source of errorsin event detection algorithms is noise. Most event detectionalgorithms assume that the analyzed signal contains only audiofrom musical sources, but that is not true for many cases.There are noises that are inherent to the audio acquisitionenvironment, like the noise of computer fans, heating systems,steps, or even crowds, in the case of live performances. Thesesources can be identified as musical notes, which may harmthe performance of the detection algorithm by increasing thenumber of false positives.

It is important to note, however, that musical note events arealso correlated to certain physical gestures that are performedby the musician. It has been shown that the combination ofaudio and visual information tends to increase the performanceof speech recognition systems [4]. The use of audio andvisual information for musical transcription, to the best of ourknowledge, was first proposed by Gillet and Richard [5] in asystem capable of integrating data acquired from a microphoneand a video camera to detect and characterize the hits on adrum kit. In this work we apply a similar approach to the livedetection of note events from a vibraphone. Gillet and Richard

show that the fusion of audio and video features can leadto better transcription results than the results obtained usingsolely audio or video. Later, Kapur et al. [6] used multiplesensor fusion to detect certain gestures in sitar performanceswhich may easily be confused with each other if only auditoryinformation is considered.

This paper proposes a multi-modal solution for the detectionof events in vibraphone performances, aimed at reducing theharmonic errors and the errors due to ambient noise whencompared to the traditional audio-only solution. Three differ-ent audio signal processing methods were tested: the Non-Negative Least Squares fitting (NNLSQ) [7], the ProbabilisticLatent Component Analysis (PLCA) [8], and the auditory-inspired multiple fundamental frequency detection methodproposed by Klapuri [9]. The computer vision technique usedin the paper relies on a video camera that is mounted on topof the vibraphone to estimate the position of the mallets inrelation to the bars. This information is combined with theaudio-based results so that positioning the mallet over a baris a necessary condition for a note to be detected.

The paper is organized as follows. In Section II, the pro-posed system and its modules are described. In Section III, theevaluation methods are described and the results are shown.Finally, in Section IV further discussions are conducted andconclusive remarks are presented.

II. SYSTEM DESCRIPTION

The system proposed in this paper relies on both audio andvideo data, which are jointly analyzed in order to detect whena musician plays a certain note. The audio data is analyzed bya real-time transcription algorithm, which consists of a signalprocessing front-end and an event detection method. The threesignal processing front-end considered were based on NNLSQ[7], PLCA [8], and the method proposed by Klapuri [9]. Theevent detection method is based on adaptive thresholding. Thevideo data is processed by a computer vision algorithm thatis responsible for detecting the position of the mallets overthe instruments’s body. Since it is necessary to hit a certainbar with the mallet in order to produce sound, the informationon the location of the mallet is used to inhibit the detectionof notes that are clearly not being hit by the mallet at thatmoment. This Section is divided in two parts. In the first one,the computer vision algorithm and the hardware requirementsfor the video camera are described. In the second one, thedifferent audio analysis algorithms are described, as well ashow the audio and visual information is integrated.

A. Computer vision

In this work, computer vision techniques are used to trackthe position of the percussionist’s mallet tips relative to thevibraphone bars. This technique allows us to determine whichbars of the vibraphone are covered by the mallets and therefore“active”. The assumption is that if the mallets are covering acertain bar, then it is likely that the pitch associated with thatbar will be heard. Of course there is also the possibility of

Fig. 2. Computer Vision Mallet Detection with a) blob detection and b) avirtual scaled copy of the vibraphone.

the mallet covering a bar without the note being played as theperformer moves the mallets across the instrument.

Using a VGA webcam (in this work, we used the onethat is integrated in the Microsoft Kinect) mounted abovethe vibraphone and computer vision libraries [10], we areable to track the vibraphone performer’s mallet tips basedon their color. This process involves filtering out everythingexcept a desired color range, and then performing contourdetection on the filtered image. The color filtering processis shown in Figure 3. The color image is first separatedinto Hue, Saturation, and intensity Value (HSV) components.A binary threshold is applied to each of these components,setting all values within a desired range to 1 and the rest to0. Recombining these images by performing boolean ANDoperations yields an image where nearly everything except thetips of the mallet sticks have been removed.

xbinary thres =

{1 if min < x < max0 otherwise

(2)

Contour detection is performed on the binary image usinga technique called border following [11]. This techniqueinvolves traversing edges between 0 and 1 in the binary imageand creating a sequence. If this sequence returns to it’s startingpoint then a contour has been detected. For our purposes, abounding rectangle around each contour is returned. Finally,the possible correct results are limited by area, excludingcontours that are too large or too small.

Tracking objects with blob detection algorithms returns po-sition data relative to the pixel space of the camera. However,we would like these positions to exist in a virtual spacethat also includes a 3D model of the vibraphone bars. Usingexplicit measurements, a scaleable model of the vibraphonewas created. Assuming linearity in the position output from thecamera, our mallet positions are transformed into our virtualspace using the linear normalization shown in Expression3. Currently, a calibration stage is needed to determine theposition of the vibraphone relative to the camera pixel space.

pnorm =po − pmin

pmax − pmin(3)

Although this method is simple, sensitive to lighting condi-tions, and requires manually setting the color of the mallet tips,it has given results that are accurate enough for the purposesof this paper.

Fig. 3. Computer Vision Flow Diagram

B. Machine listening

The audio analysis techniques we use are based on theassumption that audio signals resulting from the mixing ofseveral different sources are, in terms of physical measures,the sum of the signals corresponding to each individual source.Similarly, the human perception of listening to that soundis essentially the superposition of the sensations triggeredwhen listening to each individual source. Therefore, it isreasonable to assume that, to a certain extent, the phenomenonof detection of sound sources may be described using thefollowing linear model:

X = BA. (4)

In that model, B is a set of basis functions forming adictionary, that is, a set of vector representations of the sourcesto be identified, X is the representation of several measuresof the phenomenon in the same domain as B, and A is a setof activation coefficients that represent how much each basisfunction (corresponding to a source) defined in B is activein each measurement. Figure 4 shows an example activationmatrix, in which each line represents a different note and eachcolumn represents a different time frame. According to theharmonic model described in Expression 1, a particular notetends to have a stationary representation when consideringthe magnitude of its Discrete Fourier Transform (DFT). Thismeans that the magnitude spectrum is, for harmonic signals,one possible representation that enables the use of the linearmodel in equation 4.

The assumption of linearity in the harmonic model has been

Base n

um

ber

Time (s)

0 10 20 30 40 50

10

20

30

0

0.1

0.2

0.3

Fig. 4. Example activation matrix.

used successfully in several experiments, both using Non-Negative Matrix Factorization (NMF) [12], [13], in whichthe basis matrix B is obtained by unsupervised learning,and the NNLSQ approach [14], in which the basis matrix Bis obtained previously by means of supervised learning andonly the activation matrix is estimated. Although the NNLSQrequires a previous stage of training, it allows causal pro-cessing, which is essential for real-time applications. Anotherway of calculating a conceptually similar activation matrixis by means of the Probabilistic Latent Component Analysis(PLCA) [8]. This technique consists of assuming that both Band A denote probability distribution functions and executingthe Expectation-Maximization (EM) technique to obtain them.In the implementation used in this paper, the basis matrixB is obtained in a previous stage from training data and isnot adapted (supervised version). Both NNLSQ and PLCAare search algorithms that aim at minimizing the norm of theapproximation error ‖X − BA‖ so their results are similarbut are different due to the optimization process utilized.

An alternative method to obtain the activation matrix A is toperform an auditory inspired search for multiple fundamentalfrequencies, as proposed by Klapuri [9]. This search methodincorporates psycho-acoustic knowledge both to design thebasis matrix and to obtain the activation of each base. Klapuri’salgorithm is based on calculating a salience function for eachfundamental frequency candidate, which indicates how muchit is active. The most salient candidate is then subtracted fromthe spectrum and these two steps are iteratively performeduntil the desired number of notes is obtained. In this paper, thealgorithm was adapted so that the activation value for the fournotes corresponding to the first four estimated fundamentalfrequencies are equal to their saliences, and the activation forthe other notes are equal to their saliences after the removalof those four notes.

The audio processing algorithms rely on a framewise spec-trogram representation, calculated as follows. The input signalis sampled at 48 kHz and processed in discrete frames of46ms, with a 23 ms overlap. Each frame is multiplied bya Hanning window, zero-padded to twice its original lengthand is transformed to the frequency domain using the DFT,obtaining X . In order to minimize the influence of of variationin harmonic amplitudes, X is logarithmically scaled as inY = log10(1+‖X‖) [14] (the logarithmic scaling is bypassedwhen using Klapuri’s method, as it already performs spectralnormalization [9]). Finally, the frequency domain representa-tion is trimmed in frequency ignoring values ignore values

outside the frequency range in which the vibraphone operates.The basis functions are obtained by taking the spectrogram

of a recording containing a single note hit and averaging thefirst few frames. Only these few frames are used because, inthe vibraphone sounds, the harmonics decay quickly. Usingmore data from a particular note would converge to a spectro-gram that would be dominated by the fundamental frequencyof the series instead of all the harmonics. The spectrogram isthen normalized in order to have unity variance (but not zeromean). The normalization is used in order to ensure that thevalues in each row of the activation matrix A represent theenergy of the activation of the corresponding note.

For the factorization-based methods (NNLSQ and PLCA), aset of additional basis functions, called noise basis functions,are also included in the basis matrix (or dictionary). Noisebasis functions are calculated as triangles in the frequencydomain, which overlap by 50% and have center frequenciesthat start at 20 Hz and increase by one octave from one noisebase to the next one. This shape aims to give the systemflexibility in modeling background noise, specifically non-harmonic sounds, as filtered white noise. This way, back-ground noise is less likely to be modeled as a sum of basisvectors corresponding to notes, that is, background noise isless expected to affect the contents of the activation matrix.

For each incoming frame of audio, the activation vector iscalculated. and is then analyzed by an event detection methodthat yields decisions as to when notes are played. The methodworks as follows. In order to trigger the detection of a note,the corresponding value in the activation matrix must be overa fixed threshold α and its derivative (that is, an,t − an,t−1),must be over another threshold β. When a note is found, anadaptive threshold τ value is set to that level multiplied byan overshoot factor γ. The adaptive threshold decays linearlyat a known rate θ following the rule τt+1 = τt − (1 − θ). Anew note may only be detected if its activation value is greaterthan τ , in addition to the thresholding rules regarding α and β.Figure 5 shows artificial activation values and values for thedetection threshold max(α, τ). Three positions are marked,showing when false positives are avoided due to the fixedthreshold related to α, the adaptive threshold defined by γ andθ and, finally, due to the minimum derivative rule related toβ. Additionally to the thresholding related to each individual

Act

ivat

ion

Time

Fixedthreshold

Adaptivethreshold

Derivative

ActivationDetection threshold

Fig. 5. Example detection by thresholding. False positives are avoided dueto the fixed threshold, the adaptive threshold and the derivative rule.

note, the system deals with polyphony by assuming that a

certain activation level only denotes an onset if it is greaterthan a ratio φ of the sum of all activation values for that frame.

If the audio signal passes all tests described above, the esti-mated bar for the mallet using the computer vision algorithmis used as a necessary condition for detecting a particular note.With this additional condition, it is expected that octave errorsand other common mistakes regarding ghost notes can beavoided. It was observed, however, that the signal containingposition data was noisy, presenting several false negatives. Todeal with that problem, the signal was transformed in a multi-channel binary sinal, where a 1 in channel b means that themallet is detected over bar b during a certain time frame. Thelength and hop size of the frames, for this processing, areidentical to the ones used in the audio processing algorithm.After this conversion, the binary signal is filtered with a decayfilter, defined as:

y[n] =

{x[n] x[n] ≥ x[n− 1]

dx[n− 1] x[n] < x[n− 1], (5)

where the coefficient d is the decay amount. The filtered signalis thresholded at 0.5 (that is, is equal to 0 if y[n] < 0.5and equal to 1 otherwise). This smoothing process forces thesystem consider that the mallet is over a bar if it was actuallydetected there in the last few frames, inhibiting false negativesin that detection. In the next section, the objective evaluation ofthe effects of using this multi-modal approach for transcriptionare considered.

III. EVALUATION

The experiments described in this section were conductedusing audio and video recordings of an acoustic vibraphone.The acoustic data was gathered using a microphone placed inthe middle of the vibraphone. The video data was acquired bya videocamera placed on top of the instrument. The recordingsintentionally contain harmonically-related notes and heavy useof the sustain pedal. The Signal-to-Noise Ratio varied between−40 dB and 40 dB, both due to the room (heating system,computer fans, etc.), the instrument itself, and artificially-mixed crowd noise. The values for the parameter used in theexperiments described here were α = 0.05, β = 0.1, γ = 1.2,θ = 0.1, φ = 0.2 and d = 0.9. These parameters wereobtained empirically, by observing the typical values foundin the activation matrix and optimizing for better results. Themelodies in the evaluation dataset were manually annotated,providing a ground-truth for evaluation. The evaluation dataconsisted of a total of around 5 minutes of recordings, with 150annotated notes. The melodic patterns intentionally includepolyphony with harmonically-related intervals such as majorthirds, fifths and octaves. The number of simultaneous notesvaried from one, in simple monophonic phrases, to eight, whenthe sustain pedal was used. The dataset, as well as the sourcecode used are available upon request.

The transcription algorithm was executed over the evalua-tion data, yielding a series of symbols. In order to evaluatethe performance of the system, the symbols in the automatictranscription are matched to the symbols in the ground truth

using a proximity criterion [15]. This matching procedurefinds out what note in the automatic transcription was thebest attempt (considering a distance measure that accountsfor time and pitch deviation) to describe each note in theground truth. A particular note in the automatic transcriptionis considered correct if its pitch is equal to the pitch in theground truth and its onset does not deviate from the groundtruth by more than 100 ms. The time tolerance accountsfor an inevitable deviation due the framewise nature of thetranscription algorithm as well as an inherent human errorwhen the ground truth reference is built. This allows calcu-lating the Recall (R, number of correct events divided bythe total number of events in the ground truth), Precision(P , number of correct events divided by the total number ofyielded events) and, finally, their harmonic mean called theF-Measure (F = 2RP/(R+ P )).

This process was executed utilizing the three different audioanalysis algorithms (Klapuri’s, PLCA and NNLSQ). All threeof them were also tested in a configuration that does not usecomputer vision (blind ). We also consider a modification ofthe algorithm using only vision data (that is, a deaf algorithm),which works by considering an onset as soon as a mallet isplaced over a bar and an offset when it is removed from thebar. Two different pieces were used, one of them with heavyuse of the sustain pedal and another without using the sustainpedal. The F-Measure for both of them was calculated in eachtest, and the average of them is reported in Figure 6.

00.10.20.30.40.50.60.70.80.9

Klapuri PLCA NNLSQ

Ave

rage

F-M

easu

re

DeafBlind

Multimodal

Fig. 6. Average F-Measure for each modification of the proposed algorithm.

The Recall, Precision and F-Measure for each variationof the algorithm are averaged over the two teste tracks andreported in Table I.

TABLE IRESULTS FOR DIFFERENT ALGORITMS AND PIECES.

Variant Audio R P FDeaf - 0.34 0.15 0.22

Blind

NNLSQ 0.85 0.31 0.39PLCA 0.50 0.18 0.26Klapuri 0.45 0.25 0.31

Multimodal

NNLSQ 0.50 0.53 0.49PLCA 0.35 0.69 0.48Klapuri 0.48 0.66 0.48

The results above show that the use of computer vision in-formation has consistently improved the transcription accuracyof the tested systems. Also, regardless of the method used to

obtain the activation matrix A, the improvement obtained forthe results were roughly the same.

In order to evaluate the changes in the pitch errors due tothe use of the vision algorithm, a more detailed evaluation washeld. In this evaluation, each note yielded by the transcriptionsystem was related to the note that is the closest to it in theground truth, according to the same distance metric used toobtain the results above [15]. The pitch errors were classifiedas octave errors (if the same pitch class was found, but in adifferent octave), tone errors (if the pitch difference is equalto one or two semitones) and other errors (for other pitchdeviations). The fraction of errors in each of these categories,considering the blind and multimodal variants of the methodthat performed best in our experimentos – NNLSQ – as wellas the deaf approach, are shown in Figure 7.

0

20

40

60

80

100

Deaf Blind Multimodal

Err

ors

(%)

ToneOtherOctave

Fig. 7. Pitch errors for deaf and NNLSQ-based variants of the algorithm.

As it can be seen, when the multimodal approach is used,the fraction of octave errors was reduced. At the same time, itcan be seen that the fraction tone errors increases, represent-ing a performance tradeoff indicating further possibilities forimprovement.

IV. CONCLUSION

This paper has presented a method for the multi-modal realtime transcription of the vibraphone. The method relies ona causal transcription algorithm that operates in parallel witha computer vision algorithm. The transcription algorithm isresponsible to detect, in the audio signal, characteristics thatindicate note onsets. The computer vision algorithm yieldsinformation on the position of the mallets. Both are combinedso that an onset is only detected if the transcription algorithmindicates so and, at the same time, the mallet is over thecorresponding bar of the instrument.

Three different signal processing techniques were tested forthe audio analysis part of the algorithm: factorization usingNNLSQ [7] and PLCA [8], and the multiple fundamental fre-quency method proposed by Klapuri [9]. The results obtainedshow that using any of these algorithms result in roughly thesame transcription accuracy. Results also show that, regardlessof the signal processing used for audio analysis, the use ofinformation from computer vision has consistently improvedthe transcription accuracy.

The proposed system does not present enough accuracy forapplications such as performance analysis or documentation.However, it may still be used in applications that support moreerrors in the transcription, like chord recognition, automaticharmonization or creative soundscape design.

The are several interesting directions for future work. Thecomputer vision algorithm could be modified tracking al-gorithm would work in different lighting conditions – animportant factor for live performance applications – and inthree dimensions, so that the proximity between the mallet anda certain bar can be measured more accurately. We also plan toexplore how our systems can be used in applications such asautomaic score following, chord detection and harmonizationthat can take advantage of imperfect transcription results.

ACKNOWLEDGMENT

Tiago Fernandes Tavares thanks CAPES - Coordenacao deAperfeicoamento de Pessoal de Nivel Superior - for funding.The support of the National Science and Research Council ofCanada is gratefully acknowledged. The authors thank ShawnTrail for the assistance with the vibraphone and the researchas well as Steven Ness for help with the figures.

REFERENCES

[1] T.Machover and J.Chung, “Hyperinstruments: Musically intelligent andinteractive performance and creativity systems hyperinstruments,” in InProc. Intl. Computer Music Conference, 1989.

[2] C. Traube, P. Depalle, and M. Wanderley, “Indirect acquisition of instru-mental gesture based on signal, physical, and perceptual information,”in In Proceedings of the 2003 Conference on New Interfaces for MusicalExpression (NIME-03, 2003, pp. 42–47.

[3] H. F. Olson, Music, Physics and Engineering, 2nd ed. Dover Publica-tions Inc., 1967.

[4] G. Galatas, G. Potamianos, A. Papangelis, and F. Makedon,“Audio visual speech recognition in noisy visual environments,”in Proceedings of the 4th International Conference on PErvasiveTechnologies Related to Assistive Environments, ser. PETRA ’11. NewYork, NY, USA: ACM, 2011, pp. 19:1–19:4. [Online]. Available:http://doi.acm.org/10.1145/2141622.2141646

[5] O. Gillet and G. Richard, “Automatic transcription of drum sequencesusing audiovisual features,” in Acoustics, Speech, and Signal Processing,2005. Proceedings. (ICASSP ’05). IEEE International Conference on,vol. 3, march 2005, pp. iii/205 – iii/208 Vol. 3.

[6] A. Kapur, G. Percival, M. Lagrange, and G. Tzanetakis, “Pedagogicaltranscription for multimodal sitar performance,” in International Societyfor Musical Information Retrieval, september 2007.

[7] R. J. Hanson and C. L. Lawson, Solving least squares problems.Philadelphia, 1995.

[8] P. Smaragdis, B. Raj, and M. Shashanka, “Sparse and shift-invariantfeature extraction from non-negative data,” in Acoustics, Speech andSignal Processing, 2008. ICASSP 2008. IEEE International Conferenceon, 31 2008-april 4 2008, pp. 2069 –2072.

[9] A. Klapuri, “Multiple fundamental frequency estimation by summingharmonic amplitudes,” in Proc. 7th International Conference on MusicInformation Retrieval, Victoria, BC, Canada, Oct. 2006, pp. 1–2.

[10] Intel Corporation, Open Source Computer Vision Library. USA:http://developer.intel.com, 1999.

[11] S. Suzuki and K. be, “Topological structural analysis of digitized binaryimages by border following,” Computer Vision, Graphics, and ImageProcessing, vol. 30, no. 1, pp. 32 – 46, 1985. [Online]. Available:http://www.sciencedirect.com/science/article/pii/0734189X85900167

[12] N. Bertin, R. Badeau, and G. Richard, “Blind signal decompositionsfor automatic transcription of polyphonic music: Nmf and k-svd on thebenchmark,” in Acoustics, Speech and Signal Processing, 2007. ICASSP2007. IEEE International Conference on, vol. 1, april 2007, pp. I–65 –I–68.

[13] S. Phon-Amnuaisuk, “Transcribing bach chorales using non-negativematrix factorisation,” in Audio Language and Image Processing(ICALIP), 2010 International Conference on, nov. 2010, pp. 688 –693.

[14] B. Niedermayer, “Non-negative matrix division for the automatic tran-scription of polyphonic music,” in Proceedings of the ISMIR, 2008.

[15] T. F. Tavares, J. G. A. Barbedo, and A. Lopes, “Towards the evaluationof automatic transcription of music,” in Anais do VI Congresso deEngeharia de udio, may 2008, pp. 47–51.