SGN$14006)Audio)and)Speech)Processing)

)Acoustic)Phonetics)

Slides'for'this'lecture'are'based'on'those'created'by'Katariina'Mahkonen'for'TUT'course'”Puheenkäsi;elyn'menetelmät”.'

Other'sources:'QuaAeri:'DiscreteCTime'Speech'Signal'Processin'–'Principles&PracAce.'

'K.'Koppinen:'”Puheenkäsi;elyn'menetelmät”,'luentomoniste,'TTY,'''h;p://www.cs.tut.fi/courses/SGNC4010/sgn4010.pdf'

'

Acoustic)Phonetics)

•  AcousAcally,'speech'signal,'as'any'sound,'can'be'viewed'as'air'pressure'level'variaAon'

•  AcousAc'phoneAcs'studies'the'acousAc'characterisAcs'of'speech'and'their'relaAonship'to'speech'producAon'

'

'

2!

Link!Longitudinal waves: http://www.acs.psu.edu/drussell/Demos/waves/wavemotion.html!

Speech)waveform/spectrum)is)quasi$stationary)only)5)$20ms)

•  Speech'is'processed'in'short'frames'(frameCbyCframe)'

•  Length'of'the'frame/window'in'speech'processing'is'usually'10C20ms'

•  Hanning/Hamming'type'windows'are'commonly'used'

•  Remember'how'windowing'works:'

What)are)speech)signals)like?)•  In'Ame'domain'

•  Which'phonemes'have'a'lot'of'energy?'

•  How'does'the'voiced/unvoiced'difference'appear'in'the'signal?'•  How'do'plosives'look'like?'

•  In'the'frequency'domain'

•  How'does'the'voiced'nature'of'a'phoneme'appear'in'fCdomain?'

•  How'do'you'find'the'pitch'from'speech'spectrum?'

•  Which'phoneme'is'easiest'to'recognize'from'the'spectrogram?'

•  What'special'feature'is'visible'in'the'spectrum'of'a'nasal?'

•  So`ware'for'speech'signal'visualizaAon:'Audacity,'Wavesurfer,'Praat,'Rtgram'(Windows),'Baudline'(Linux)'

4!

Audacity download!

Windows: RTgram download/! Linux: Baudline download!

5!

Example sentence: ”He knew what taboos…” (arctic_b0510.wav)!

h! e! (k)n! ew! t! a! b! s!w(h)! a! t! oo!

Modeling))speech)

6!

Quatieri: Discrete-Time "Speech Signal Processing"- Principles and Practice!

Larynx)excitation))(glottis)signal))

Function)of)the)vocal)folds)

A glottis is closed when swallowing!B in voiced phonemes, the vocal chords vibrate periodically!C when whispering, airflow passes only through interarytenoid space!D in glottal fricatives (/h/), vocal folds are narrowly open !E rest/breathing position!F yawning!

What kind of signal (and spectrum) is produced from these?!

Electroglottograph)(EGG))

! Measures vocal"fold contact area!

Inverse)Oiltering)

How)does)glottal)signal)look)like?)

•  In'voiced'phonemes,'vibraAng'vocal'folds'cause'pressure'pulses'at'the'

vibraAon'rate'(fundamental'frequency'fate'F0)'(posiAon'B'above)'

•  In'unvoiced'phonemes,'the'narrow'hole'in'the'larynx'causes'a'turbulence'

in'the'airstream'from'the'lungs'(posiAon'C).'

Waveshapes)of)periodic)glottal)signal)

Spectra)of)periodic)glottal)signal)

13!Aiheesta University of New South Walesin (Australiassa) sivuilla!

http://www.phys.unsw.edu.au/jw/glottis-vocal-tract-voice.html!

Other)sources)of)sound)energy)in)speech)

1.  A'constricAon'(narrow'point)'in'the'vocal'tract'causes'a'

turbulence'in'the'air'stream'that'passes'through'

•  Difference'to'whispering:'here'the'turbulence'occurs'in'the'vocal'tract'and'not'in'the'gloes'

2.  Vocal'tract'is'closed'to'build'up'pressure'which'is'then'

released'and'the'air'”explodes”'out'(plosives)'

14!

Modeling)speech)–)inOluence)of)vocal)tract)on)speech)spectrum)

15!

Quatieri: Discrete-Time Speech Signal Processing - Principles and Practice!

Amazing grace (overtone singing)!

Spektrograms of overtone singing!

Formants:)resonances)of)vocal)tract)•  The'most'important'characterisAc'of'the'vocal'tract'are'its'resonances'(formants)'

•  Due'to'standing'waves'in'the'vibraAng'air'column'

•  Formants'(F1,'F2,'...)'can'usually'be'seen'in'the'spectrum'as'boosted'frequency'regions'

•  In'addiAon'to'frequency,'a'formant'is'characterized'by'its'intensity'and'bandwidth'

•  Different'vocal'tract'configuraAons'correspond'to'different'formant'frequencies'"'all'vowels'can'be'classified'based'on'formants'

''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''spectrum'of'phoneme'/a/'

'

'

''

'

''

'

'

'

16!

Standing waves!

Vowels)$)formants)

17!

Mathematical)modeling)of)the)vocal)tract)•  CalculaAng'the'vocal'tract'resonances'based'on'the'shape'of'the'vocal'tract'is'analyAcally'intractable'(numerical'soluAons'

exist)'

'C'Should'take'into'account''

•  Different'larynx'excitaAons,''•  TimeCvarying'and'locaAonCdependent'changes'in'the'

vocal'tract'shape'

•  Nasal'tract'opening/closing,''•  Sound'radiaAon'from'the'lips,''

•  Energy'losses,''•  Turbulences''•  etc.'

''

•  However'by'studying'simplified'models'we'can'gain'a'fair'

amount'of'understanding'of'speech'producAon'

'

'

18!

Acoustics)of)simple)tubes)

19!

•  Resonances'of'the'vocal'tract'are'due'to'standing'waves'in'the'vibraAng'air'column'(similar'to'e.g.'wind'instruments)'

•  In'a'tube'of'uniform'crossCsecAonal'area,'the'wavelengths'λ'of'standing'waves'are''

•  SubsAtuAng'the'typical'vocal'tract'length'(male'0.17m,'female'0.15m),'the''frequencies'of'the'resonances'would'be'500Hz,'1500Hz,'2500Hz,…'

•  In'vocal'tract,'the'crossCsecAonal'area'varies'and'thus'resonance'frequencies'vary,'but'as'a'rule'of'thumb,'there'is'roughly''1'resonance'per'1'kHz'

Acoustics)of)simple)tubes)

20!

•  The'acousAcs'of'a'uniform'tube'can'be'solved'exactly'and'it'helps'us'in'the'following'(notaAon'x,'l,'S'on'the'previous'slide)'

•  AcousAcally'interesAng'variables'are'parAcle'velociAes'v(x,t)'in'the'tube'at'point'x'and'Ame't'and'pressure'p(x,t)'

•  For'simplicity,'we'assume'planar'pressure'waves'that'travel'along'the'tube.'For'convenience'we'use'volume'velocity'u(x,t)'instead'of'parAcle'velocity:'u(x,t)'='Sv(x,t)'

•  The'relaAonship'between'pressure'and'volume'velocity'is'governed'by'soCcalled'wave%equa(ons:'''''''where'ρ'denotes'atmospheric'pressure'and'c'is'speed'of'sound'

−∂p∂x

=ρS∂u∂t

−∂u∂x

=Sρc2

∂p∂t

! Intuition: if particle at point x is not moving but pressure is higher at its right side, pressure difference causes the particle accelerate to left.!

! Intuition: if pressure at point x is zero but particle velocity is higher on the right hand side, particles ”pile up” at point x and pressure increases.!

Solution)to)the)wave)equation)

21!

•  It'is'quite'easy'to'see'that'given'an'arbitrary'funcAon'f(y),'the'following'is'a'soluAon'to'the'wave'equaAon:''''u(x,t)'='f(t'–'x/c)'''''p(x,t)'='(ρc/S)'f(t'–'x/c)'which'is'simply'a'forward'traveling'wave'at'speed'c.'That'can'be'verified'by'subsAtuAng't!'t+1'and'x!'x+c'and'noAng'that'the'funcAon'gets'the'same'values'(wave'travels'for'one'sec.).'

•  Similarly,'a'backward'traveling'wave'is'a'soluAon'to'the'wave'equaAons.''

•  Alltogether,'we'can'write'the'soluAon'in'generic'form'as'''''u(x,t)'='f(t'–'x/c)'–'b(t'+'x/c)'''''p(x,t)'='(ρc/S)''['f(t'–'x/c)'+'b(t'+'x/c)']'where'f'and'b'are'arbitrary'forward'and'backward'traveling'waves,'respecAvely'

Modeling)the)vocal)tract)with)simple)tubes)

•  Vocal'tract'is'straightened'and'modeled'using'slices'with'constant'length'and'uniform'crossCsecAonal'area'

22!On the shape of the vocal tract “tube”!http://www.davidmhoward.com/voiceSoundModifiers.htm!

ReOlection)of)the)pressure)wave)

23!

nf

nn fknn fk )1( −

•  When two simple tubes are joined, reflections occur at the boundary!!

•  Reflection coefficient kn indicates how large part of "the volume-velocity wave traveling from a tube to the next is reflected back (tube cross-sectional areas Sn and"Sn+1 ):!

!!

1

1

+

+

+

−=

nn

nnn SS

SSk

ReOlection)of)the)pressure)wave)

•  Areas'are'posiAve,'therefore'91'<'kn'<'1'•  If'Sn+1'='0'','then''kn''=''1','and'the'wave'is'reflected'back'as'it'is'•  If'Sn+1'is'large,'kn''≈'C1','and'the'wave'is'reflected'in'its'enArety,'but'in'opposite'phase'

•  If'Sn'=Sn+1','no'reflecAon'occurs'

24!

Reflection of waves!1

1

+

+

+

−=

nn

nnn SS

SSkwww.acs.psu.edu/drussell/Demos/reflect/reflect.html!

Modeling)wave)reOlections)in)z$plane)–))lattice)structure)(Kelly$Lochbaum)structure))

25!-> length of one model slice = 340 m/s / fs!

•  fn'is'forwardCtraveling'sound'wave'in'the'tube'and'bn'is'backwardCtraveling'wave'

•  Let’s'model'the'wave'propagaAon'

and'reflecAon'using'the'KellyC

Lochbaum'laece'structure'shown'

in'the'figure'on'the'right'

•  Let’s'sample'the'behaviour'of'the'

structure'so'that'the'wave'is'

delayed'by'one'Ame'unit'(zC1)'when'it'travels'the'length'of'one'tube'

secAon'

Kelly$Lochbaum)equations)•  Wave'behaviour'can'be'described'using'the'following'

equaAons.'

•  From'the'figure'we'

obtain:'

26!

)()()1()( 11

1 zbkzzfkzf nnnnn +−

+ −−=

11

2 )()1()()( −+

− ++= zzbkzzfkzb nnnnn

Kelly$Lochbaum)equations)'

•  Let’s'solve'fn+1(z)'and'bn+1(z)'as'a'funcAon'of'fn(z)'and'bn(z)''•  SoluAon'in'matrix'format:''

27!

!"

#$%

&

!!!!

"

#

$$$$

%

&

++−

+−

+=!

"

#$%

&−

−

+

+

)()(

11

11)()(

1

1

1

1

zbzf

kz

kzk

kzk

kz

zbzf

n

n

nn

n

n

n

n

n

n

!"

#$%

&=!

"

#$%

&

+

+

)()(

)()(

1

1

zbzf

Hzbzf

n

nn

n

n

Vocal)tract)model)using)the)lattice)structure)

•  We'obtain'a'discreteCAme'representaAon'for'the'tubeCsegment'

model'of'the'vocal'tract'by'concatenaAng'laece'elements'28!

Lattice)structure)

•  The'model'has'been'found'to'work'sufficiently'well'also'with'

the'simplified'assumpAon'b0=0'ja'bN=0'•  Transfer'funcAon'of'the'enAre'laece'structure'is'obtained'as:'

•  So:'

'

29!

!"

#$%

&==!

"

#$%

&=!

"

#$%

&=!

"

#$%

&−

−

−−

−

−

)()(

......)()(

)()(

)()(

0

001

2

21

1

1

zbzf

HHHzbzf

HHzbzf

Hzbzf

NNN

NNN

N

NN

N

N

!"

#$%

&Η=!

"

#$%

&=!

"

#$%

&=!

"

#$%

&− 0

)()(

)()(

...)()(

0)(

0

001

zGz

zbzf

HHHzbzfzS

NNN

N

The)Oilter)used)to)model)the)vocal)tract)'

•  We'observe'that'the'filter'H(z)'is'of'allCpole'type:'

•  Thus'the'vocal'tract'is'modeled'using'the'aboveCdescribed'allCpole'

structure,'aka'autoregressive'(AR)'model.'That'is'system'whose'

transfer'funcAon'is'of'the'form'

•  The'next'lecture'will'discuss'a'mathemaAcal'technique'called'

LINEAR'PREDICTION'(LP)'that'allows'us'to'determine'the'

coefficients'of'the'funcAon'A(z).'The'linear'predicAon'can'in'pracAce'

be'computed'using'a'(fast)'LevinsonCDurbin'algorithm.'

'

'

30!

)(1)(zA

z =Η

But)before)that…)

Some vocal talent!http://www.youtube.com/watch?v=ZxcnloCzxq4!