audio morphing for percussive sound generation

Audio MorphingProposed Algorithm

ResultsConclusion

Audio Morphing for Percussive Hybrid SoundGeneration

Andrea Primavera1, Francesco Piazza1 and Joshua D. Reiss2

1 A3lab - DIBET - Universita’ Politecnica delleMarche - Ancona - ITALY

2 C4DM - Queen Mary University of London -London - UK

Andrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 1/21


ResultsConclusion

IntroductionState of the Art

Audio Morphing is typically employed to accomplish two differenttasks:

• Obtaining a smooth transition between two sounds;

• Generating a new intermediate sound which has thecharacteristics of the two given samples (e.g., timbre andduration).

Topic

An automatic audio morphing technique applied to percussive musicalinstruments has been developed in this work.

Proposed Innovation

Creating new hybrid percussive sounds perceptually interesting (e.g.,morphing among different brands of the same instrument).

AIM



ResultsConclusion

IntroductionState of the Art

During the last decades several algorithms have been presented to morphaudio signals: linear interpolation, sinusoidal plus residual [1] [2] andspectral envelope using various features [3].

One of the first approaches used toimplement audio morphing.

y(t) = αs1(t) + (1− α)s2(t)

PRO: Low computational cost;

CONS: When the timbre of the in-

put signals is slightly different, both

the references could be individually

perceivable.

Linear Interpolation (LI) Based on harmonic modeling [4] [5]:

s(t) =N∑

n=1

An(t)cos[θn(t)] + e(t)

The morphed sound is obtained in-

terpolating the fundamental frequen-

cy, the harmonics timbre, and the re-

sidual envelope.

PRO: Capability of morphing sounds

with different timbre;

CONS: Higher computational co-

st than LI, some problems with

percussive sounds.

Sinusoidal Plus Noise (SMS)



ResultsConclusion

Proposed AlgorithmPreProcessingProcessing

An automatic audio morphing algorithm employed to generate newhybrid percussive sounds is presented here. It is based on:

• PreProcessing (frequency domain);

• Processing exploiting linear interpolation (time domain).

Proposed Algorithm

Fig.1 Block diagram of the proposed morphing algorithm: yellow indicates thefrequency domain while blue represents the time domain.



ResultsConclusion


The PreProcessing operation is performed on both the signal referencesin order to obtain hybrid sounds that change in a linear fashion when theinterpolation factor varies linearly.

It allows to align the samples before executing the morphing exploitingcross-correlation.

1. Time Align



ResultsConclusion



• The AR model is employed instead of the ADSR;

• Attack and release are determined using the Amplitude CentroidTrajectory [6] [7] approach and evaluating the T60 [8] for eachaudio sample.

Fig. 2: Time evolution of the spectralcentroid for a percussive sample.

SpectralCentroid [t] =

∑Mb=1 fb[t]ab[t]∑M

b=1 ab[t]

where:

f is the frequency and a is the amplitude of

the b up to M bands, evaluated using FFT.

2. AR Model Analysis



ResultsConclusion



The release portions are time stretched or compressed as a functionof the interpolation factor value:

tr = αtr1 + (1− α)tr2

3. Time Stretching



ResultsConclusion


The Processing operation is executed on the preprocessed signalreferences using the linear interpolation approach.

Since percussive sounds are typically characterized by a noisy spectrumand “similar” timbre this approach is preferable to additive synthesis.

Linear Interpolation



ResultsConclusion

ResultsAnalysis of Time and Spectral Features1 Listening Test (MOS)2 Listening Test (MDS)

Several tests have been carried out to evaluate the effectiveness of theproposed approach through objective and subjective comparisons.

Real recordings of drum kitshave been employed using the

BFD2 sound library [9].



ResultsConclusion



Spectral centroid evolution and release time have been evaluated as afunction of the interpolation factor.

Analyzed algorithms:

• Linear Interpolation (LI);

• Linear Interpolation with Preprocessing (LIP);

• Sinusoidal plus Residual (SMS).

Morphing examples:

• Different kind of snare drums (i.e, electronic and classic snare);

• Toms of different size and brand;

• Hi-hats of different brand.

Analysis of Time and Spectral Features (Objective Measure)



ResultsConclusion



In order to evaluate the audio perception of the produced hybrid soundstwo different listening tests [10] have been performed:

• Mean Opinion Score (MOS) −→ Audio quality, estimatedinterpolation factor;

• MultiDimensional Scaling (MDS) −→ Perceptual space.

Analyzed algorithms:

• Linear Interpolation (LI);

• Linear Interpolation with Preprocessing (LIP);

• Sinusoidal plus Residual (SMS).

Morphing examples:

• Different kind of snare drums (i.e, electronic and classic snare);

Listening test (Subjective Measure)



ResultsConclusion


SNARE TOM HI-HAT

0 0.2 0.4 0.6 0.8 1

0.8

1

1.2

1.4

1.6

1.8

Interpolation Factor

Re

lea

se T

ime

(se

c)

LILIPSMS

0 0.2 0.4 0.6 0.8 12.8

3

3.2

3.4

3.6

3.8

4


Re

lea

se T

ime

(se

c)

LILIPSMS

0 0.2 0.4 0.6 0.8 1

1.4

1.6

1.8

2

2.2

2.4

2.6

2.8


Re

lea

se T

ime

(se

c)

LILIPSMS

Fig. 3 Release time as a function of different interpolation factor.

0 0.2 0.4 0.6 0.8 1

2000

2500

3000

3500


Sp

ect

ral c

en

tro

id (

Hz)

LILIPSMS

0 0.2 0.4 0.6 0.8 1820

840

860

880

900

920

940

960

980


Sp

ect

ral c

en

tro

id (

Hz)

LILIPSMS

0 0.2 0.4 0.6 0.8 18000

8500

9000

9500


Sp

ect

ral c

en

tro

id (

Hz)

LILIPSMS

Fig.4 Spectral centroid evolution as a function of different interpolation factor.

• Release time changes linearly with the change of the interp. factor;

• The spectral centroid does not seem affected by the preprocessing.

Considerations



ResultsConclusion


Based on MOS, the estimated interpolation factor and the perceivedaudio quality for LI, LIP and SMS are evaluated in the morphing betweena classic and electronic snare.

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1


Est

. In

terp

. F

act

or

LILIPSMS

Fig. 5 Estimated interpolation factor.

LI LIP SMS

Quality 0.90 0.88 0.10STD 0.06 0.07 0.08

Tab. 1 Percived audio quality expressed ina scale from 0 to 1 (0 bad - 1 excellent).

• For the proposed approach (LIP) the estimated interpolation factorchanges roughly linearly with the change of the interpolation factor.

• For linear interpolation (LI) the factor 0.5 was perceived as near 0.8.

• Samples generated using SMS sound unnatural, this confirm theharmonic models limitation in the reproduction of percussive sound.

Considerations



ResultsConclusion


Based on MDS [11] [12], the test aims to recreate the correspondingperceptual spaces of the hybrid sound generated in morphing between aclassic and electronic snare with LI and LIP.

−0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

12

3

45

678

1 Dimension

2 D

ime

nsi

on

Fig.6 MDS analysis obtained with LI.

−0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

1

2 3 45

67

8

1 Dimension

2 D

ime

nsi

on

Fig.7 MDS analysis obtained with LIP.

LI LIP

CORR 0.986 0.980

P-VAL 7.2*10−6 1.9*10−5

Tab. 2 Correlation between firstdimension and release time.

LI LIP

CORR 0.66 0.77P-VAL 0.07 0.02

Tab. 3 Correlation between seconddimension and audio quality.



ResultsConclusion


Fig.8 Correlation between firstdimension and release time for LI and

LIP.

The curves have been shifted on

the y-axis in order to enhance the

readability.

• High correlation between therelease time and themovement along the firstdimension.

• Release time is an importantfactor in the evaluation ofpercussive audio morphingalgorithms.

• Although the correlationbetween the seconddimension and the audioquality is not zero, the valuesobtained are not sufficient toestablish a strong relationshipbetween them.

Considerations



ResultsConclusion

ConclusionFuture WorksQuestionsBibliography

In conclusion:

• An automatic audio morphing technique applied to percussivemusical instruments has been presented in this work.

• The presented technique is based on preprocessing of the sound andlinear interpolation time domain.

• Different tests have been carried out in order to evaluate theproposed approach, in terms of subjective evaluation and objectivemeasures comparing it with the previous state of the art.

• Objective analysis confirms that release time changes linearly withthe change of the interpolation factor while spectral centroid doesnot seem affected by the preprocessing.

• Listening tests confirm a linear evolution in the perception of thehybrid sounds using the presented approach.

• MDS multidimensional scaling suggests that release time is the mostimportant feature in the perception of hybrid percussive sounds.



ResultsConclusion


Future work will be oriented toward:

• Further investigation of the proposed approach analyzing theobtained performance for more dissimilar sounds.

• Real time implementation of the approach, considering an embeddedplatform.

• Refinement of the preprocessing phase, extending it with the ADSRmodel.



ResultsConclusion


QUESTIONS?



ResultsConclusion


M. H. Serra, D Rubine, and R. Dannenberg,“Analysis and synthesis of tones by spectral interpolation,”J. Audio Eng. Soc., vol. 38, no. 3, pp. 111–128, March. 1990.

N Osaka,“Timbre interpolation of sounds using a sinusoidal model,”in Proc. Int. Computer Music Conference, Banff, Canada, Sep 1995,vol. 34.

M. Slaney, M. Covell, and B. Lassiter,“Automatic Audio Morphing,”in Proc. IEEE International Conference on Acoustics, Speech andSignal Processing, Atlanta, May. 1996.

X. Serra and J. Smith,“Spectral Modeling Synthesis: A Sound Analysis/Synthesis SystemBased on a Deterministic Plus Stochastic Decomposition,”J. Computer Music, vol. 14, no. 4, pp. 12–24, 1990.



ResultsConclusion


F. Boccardi and C. Drioli,“Sound morphing with Gaussian mixture models,”in Proc. International Conference on Digital Audio Effects),Limerick, Ireland, December 2001.

John Hajda,“A New Model for Segmenting the Envelope of Musical Signals: TheRelative Salience of Steady State versus Attack, Revisited,”in Proc. 101th Audio Engineering Society Convention (AES’96), LosAngeles, USA, Nov. 1996.

J.J. Burred, X. Rodet, and M. Caetano,“Automatic Segmentation of the Temporal Evolution of IsolatedAcoustic Musical Instrument Sounds Using Spectro-Temporal Cues,”in Proc. International Conference on Digital Audio Effects(DAFX’10), Graz, AU, Sep. 2010.

Sabine, W.C.,“Collected Papers on Acoustics,” 1964, reprinted by Dover, NewYork.



ResultsConclusion


“BFD2 version 2.0.1 Acoustic Drum Production Environment,fxpansion,” .

ITU-R BS. 1534,“Method for subjective listening tests of intermediate audio quality,”2001.

D. Williams and T. Brookes,“Perceptually motivated audio morphing: warmth,”in Proc. of 128th Audio Engineering Society Convention (AES’10),London, UK, May. 2010.

A. Zacharakis and J. Reiss,“An additive synthesis technique for independent modification of theauditory perceptions of brightness and warmth,”in Proc. of 130th Audio Engineering Society Convention (AES’11),London, UK, May. 2011.


audio morphing for percussive sound generation

Education

toimplement audio

artaudio morphing

percussive musicalinstruments

new intermediate sound

etthe morphed sound

timbre andduration

linear interpolation

harmonics timbre