virtual acoustics and 3d sound in multimedia signal processing

8/10/2019 Virtual Acoustics and 3D Sound in Multimedia Signal Processing

1/189

Helsinki University of Technology Laboratory of Acoustics and Audio Signal Processing

Espoo 1999 Report 53

VIRTUAL ACOUSTICS AND 3-D SOUND IN

MULTIMEDIA SIGNAL PROCESSING

Jyri Huopaniemi


2/189


3/189

Helsinki University of Technology Laboratory of Acoustics and Audio Signal Processing

Espoo 1999 Report 53

VIRTUAL ACOUSTICS AND 3-D SOUND IN

MULTIMEDIA SIGNAL PROCESSING

Jyri Huopaniemi

Dissertation for the degree of Doctor of Science in Technology to be presented with due permission for

public examination and debate in Auditorium S4, Department of Electrical and Communications

Engineering, Helsinki University of Technology (Espoo, Finland) on the 5th of November, 1999, at 12

o'clock noon.

Helsinki University of Technology

Deparment of Electrical and Communications Engineering

Laboratory of Acoustics and Audio Signal Processing

Teknillinen korkeakoulu

Shk- ja tietoliikennetekniikan osasto

Akustiikan ja nenksittelytekniikan laboratorio


4/189

Helsinki University of Technology

Laboratory of Acoustics and Audio Signal Processing

P.O.Box 3000

FIN-02015 HUT

Tel. +358 9 4511

Fax +358 9 460 224

E-mail [email protected]

Jyri Huopaniemi

Cover picture of MarienkircheErkki Rousku

ISBN 951-22-4706-2

ISSN 1456-6303

Libella Oy

Espoo, Finland 1999


5/189


6/189


7/189


8/189


9/189


10/189


11/189


12/189


13/189


14/189


15/189


16/189


17/189


18/189


19/189


20/189


21/189


22/189


23/189


24/189


25/189


26/189


27/189


28/189

and PropertiesSource

- source directivity

- modeling of

- artificial reverb

- modeling of

acoustic spaces

spatial hearing

Room Geometry

. speech and sound synthesis

MODELING

Multichannel

MEDIUM

SOURCE

RECEIVERRoom

DefinitionHRTF

Database

Listener

ModelingSource

Modeling

Modeling

.

REPRODUCTION

DEFINITION

Binaural

loudspeaker

absorption

HRTFs.. simple models

. propagation

headphone /

- natural audio

- synthetic audio


29/189


30/189


31/189

Hl Hr Hi Hi

Hc Hc

xm

y l y lyr yr

xl xr


32/189


33/189

DIRECTRAY-BASED

COMPUTATIONAL

MODELING

ELEMENT

METHODS

MODELING

MODELING OF ROOM ACOUSTICS

INDIRECT

MODELINGMODELING

WAVE-BASED

MODELING

STATISTICAL

ACOUSTIC-SCALE

MODELING

MODELING

METHOD

DIFFERENCE

METHODS

RAY-

TRACING

IMAGE-SOURCE


34/189

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 21

0.5

0

0.5

1

Amplitude

Direct Sound

Early Reflections (< 80100 ms)

Late Reverberation (RT60 ~2.0 s)

Left Channel

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 21

0.5

0

0.5

1

Time / s

Amplitude

Right Channel


35/189


36/189

RAY TRACINGIMAGE-SOURCE

AURALIZATION

DIFFERENCEMETHODMETHOD

NON-REAL-TIME ANALYSISREAL-TIME SYNTHESIS

ACOUSTICAL ATTRIBUTES

REVERBERATION

ARTIFICIAL LATEDIRECT SOUND AND

EARLY REFLECTIONS

OF THE ROOM

MEASUREMENTS

ROOM GEOMETRY

MATERIAL DATA


37/189


38/189

T0

TN

( )z

( )zlate reverberationunit, R

directionalfiltering(ITD+HRTF)

( )z

crosstalkcancelingC

0F ( )z

m

x ( )n

yr

ly

. . .

. . .

. . .

. . .

. . .

. . .

z

z( )1

absorption,air and materialsource directivity,

1/r attenuation

F ( )

sound input

zT ( )1

z zz

direct sound and early reflections

out(left)

out(right)

(optional)

binaural outputlate reverberation

( )

z

n( )

n( )

-d -d -d

FN

0 1 N


39/189

Physical modelingGuitar synthesisDouble bass synthesisFlute synthesis

Conductor gestureanalysis

Animation &visualization

User interface

Image sourcecalculation

Auralizationdirect sound and early reflectionsbinaural processing (HRTF)

diffuse late reverberation

AscensionMotionStar

DisplaySynchr

onizat

ion

Midic

ontrol

Instrument audio(ADAT, Nx8 channels)

loudspeakersor with

Motiondata

Listenermovements

Conductor Listener

MIDISynthesizerfor drums

Optional ext.audio input

Listenerposition data Binaural reproduction

either with headphones


40/189


41/189


42/189


43/189


44/189


45/189


46/189

1

2

^y (n)

M

Sound

source

D (z)

1

My (n)

y (n)1

y (n)2Sound y (n)2

My (n)

y (n)

sourceD (z)

D (z)


47/189

0

5

10

0

50

100150

-20

-15

-10

-5

0

Frequency (kHz)Azimuth Angle ()

Magnitude(dB)


48/189


49/189


50/189

102

103

104

050

100150

30

25

20

15

10

5

0


51/189


52/189


53/189


54/189

Directivity of BK4128 dummy head, mouth opening mic position

30

210

60

240

90

270

120

300

150

330

180 00 dB6 dB12 dB18 dB

125 Hz

250 Hz

500 Hz

2000 Hz

4000 Hz


55/189

Directivity of BK4128 dummy head, mouth transducer mic position

30

210

60

240

90

270

120

300

150

330

180 00 dB6 dB12 dB18 dB

125 Hz

250 Hz

500 Hz

2000 Hz

4000 Hz

Spherical head model directivity

30

210

60

240

90

270

120

300

150

330

180 00 dB6 dB12 dB18 dB

125 Hz

250 Hz

500 Hz

2000 Hz

4000 Hz


56/189


57/189


58/189


59/189


60/189


61/189

2

1

0

Material 1. Order: a=1, b=1

2

1

0


2

1

0


7

6

5

4

3


2

1

0


2

1

0


2

1

0


Magnitude(dB)

2

1

0


Magnitude(dB)

3

2

1

0


3

2

1

0


2

1

0


6

4

2

0


2

1

0


10

5

0


200 1000 10000

15

10

5

0


Frequency (Hz)

200 1000 10000

10

5

0


Frequency (Hz)


62/189


63/189


64/189

M

AI

material absorption

source directivity

air absorption

distance attenuation

0-N( )zT ={

0-N

1-Nz( )

z( )

D0-N

z( )


65/189

0-N( )zF

ITD0-N,L

( )z

minimum-phaseHRTFs

interaural timedifference

ITD0-N,R

( )z

={H

0-N,L( )z H

0-N,R( )z


66/189


67/189


68/189


69/189

USER INPUTUSER INPUTDADATTAA

HRTF Database = [0:10:350]

= [90:10:90]

AudioInput

ITDTable

Azimuth Elevation

CoefficientInterpolation

DLl

DLr

hl,i(n,,)

hr,i(n,,)

LoudspeakerListening

HeadphoneListening

OUTPUTOUTPUT

Cross-talk

Canceling


70/189


71/189


72/189

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

x 103

2

1

0

1

2

Impulse response, azim= 40, id=2

Amplitude/origi

nal

Left

Right

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

x 103

2

1

0

1

2

Time / s

Amplitude/min.p

hase

Left

Right


73/189

103

104

25

20

15

10

5

0

5

10

15

20

25Magnitude response, azim= 40, id=2

Frequency / Hz

Magnitude/dB

Left

Right

Original ROriginal LMin.phase RMin.phase L

A A'

asin

a

a

Left

Right


74/189


75/189

103

104

30

20

10

0IPD, azim= 40, id=2

Phase(unwrapped)

/rad

Original IPDMin.phase + linear excess approx.

103

104

1

0.8

0.6

0.4

0.2

0x 10

3 ITD, azim= 40, id=2

Frequency / Hz

ITD/s

Original ITDMin.phase + linear excess approx.LF model (01.5 kHz)HF model (1.520 kHz)


76/189

Frequency-independent modelKuhn low-freq modelKuhn high-freq modelMeasured ITDs (33 subjects)

0 50 100 150 200 250 300 350

-8

-6

-4

-2

0

2

4

6

8

x 10-4

Azimuth angle (deg)

ITD(s)

ITD approximations vs. measurements, a=0.0875 m


77/189

0 50 100 150 200 250 300 350

5

0

5

x 104

Azimuth (degrees)

ITD(s)

ITD measured at different elevations for a human subject

30 deg

15 deg

0 deg

60 deg

90 deg

0 50 100 150 200 250 300 350

5

0

5

x 104

Azimuth (degrees)

ITD(s)

Modeled elevationdependent ITD

30 deg

15 deg

0 deg

60 deg

90 deg


78/189


79/189


80/189


81/189

0 50 100 150 200 250 300 350

5

0

5

x 104 ITD for distances 1.9 and 0.7 m

Azimuth (deg)

ITD(s)

HRTF Farfield

HRTF Nearfield

0 50 100 150 200 250 300 350

5

0

5

x 104

Azimuth (deg)

IT

D(s)

Spherical FarfieldSpherical Nearfield


82/189

102

103

104

-30

-25

-20

-15

-10

-5

0

5

10

Frequency (Hz)

MagnitudeResponse(dB)

Spherical head HRTF for a source at 0.3 m distance

0

50

70

90

100

110

130

140

150

180

170

102

103

104

-25

-20

-15

-10

-5

0

5

10

Frequency (Hz)

MagnitudeResponse(dB)

Spherical head HRTF for a source at 2.0 m distance

0

90

100

110

130

140

150

180

160

170


83/189

102

103

104

10

0

10

20

30

Magnitude(dB)

ILD comparison, mean across 9 subjects, azi=0 deg

HRTF Farfield

HRTF NearfieldSpherical FarfieldSpherical Nearfield

102

103

104

10

5

0

5

10

Frequency (Hz)

Magnitu

de(dB) HRTF Fartonearfield gain

Spherical Fartonearfield gain

102

103

104

10

0

10

20

30

Magnitude(dB

)


102

103

104

10

5

0

5

10

Frequency (Hz)

Magnitude(dB)


84/189

102

103

104

10

0

10

20

30

Magnitude(dB)


102

103

104

10

5

0

5

10

Frequency (Hz)

Magnitu

de(dB)

102

103

104

10

0

10

20

30

Magnitude(dB

)


102

103

104

10

5

0

5

10

Frequency (Hz)

Magnitude(dB)


85/189


86/189


87/189

102

103

104

100

101

102

Frequency (Hz)

Resolution(Q-value)


88/189


89/189

102

103

104

80

70

60

50

40

30

20

10

0

10

Frequency (Hz)

RelativeMagnitude(dB)

HRTF: person: 1, azimuth: 0, elevation: 0, right ear

256tap origBark

ERB

1/3 oct

1/10 oct

Cepstral


90/189

0 10 20 30 40 50 60 70 80 906

5

4

3

2

1

0

1

Time (samples)

Amplitude

HRIR: person: 1, azimuth: 0, elevation: 0, right ear


91/189

102

103

104

100

101

102

Frequency (Hz)

RelativeWeight

ERB weightingBark weighting


92/189

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

=-0.8

=-0.

6

=-0.

4

=-0.2

=0.0

=0.2

=0.4

=0.6

=0.8

Normalized original frequency

Normalizedwarpedfrequen

cy


93/189


94/189

102

103

104

30

20

10

0

10

20

30

40

50

60

Frequency (Hz)


HRTF: person: 1, azimuth: 20, elevation: 0, right ear

OriginalDf equalized


95/189


96/189


97/189


98/189


99/189


100/189


101/189

0

1

2

etc.

in

out

(a)

z-1

0

1

2

z -1

z-1

etc.

in

out

+

+

(b)

D (z)1

D (z)1

x 0

x 1

x 2

z-1

z-1

z-1

1

2

0

1

2

etc.

ina) b)

out+

+

+

x 0

x 1

y 1

y 2

y 3

x 2

z -1

z -1

z -1

1

2

3

0

1

2

etc.

g=1/in out

+

+

+

0


102/189


103/189


104/189


105/189

0 01 1

01

0 0

0 0

1 1

1 1

0 0

0 0

1 1

1 1

01

h

D

C

Bh

h

0 0

0 0

1 1

1 1

Ah

Eh

0 0

0 0

1 1

1 1

0 0

0 0

1 1

1 1

0

0

1

1


106/189


107/189

noise

low passfiltering

GTFB

GTFB

rectificationhalf wave

Spectrum

Spectrum

Left LL

Right LL

spectrum

ITD

pink

HRTF

HRTFL

R

LL

LL

LL

IACC

IACC

IACC

LL

LL

LL


108/189


109/189


110/189


111/189

Minimum-phase OriginalFIR, window: boxcar, order: 40WFIR, lambda=0.65, window: boxcar, order: 13IIR, design: Prony, order: 20

WIIR, lambda= 0.65, design: Prony, order: 10WIIR2, lambda= 0.7233, design: Prony, order: 20 BMT IIR, order: 10

102

103

104

20

30

40

50

60

70

80

90

Frequency (Hz)


Modeling of DF-equalized minimum-phase KEMAR HRTFs: elev=0, azi=30

Minimum-phase OriginalFIR, window: boxcar, order: 80WFIR, lambda=0.65, window: boxcar, order: 27

IIR, design: Prony, order: 40WIIR, lambda= 0.65, design: Prony, order: 20WIIR2, lambda= 0.7233, design: Prony, order: 40

102

103

104

-40

-30

-20

-10

0

10

20

Frequency (Hz)


Modeling of minimum-phase B&K4100 HRTFs: elev=0, azi=30


112/189


113/189


114/189

102

103

104

-80

-70

-60

-50

-40

-30

-20

-10

0

Frequency (Hz)


IIR approximation, azimuth=135, elevation 0, left ear

Original 256-tap FIR

48

36

24

12


115/189

102

103

104

-80

-70

-60

-50

-40

-30

-20

-10

0

Frequency (Hz)


WIIR approximation, azimuth=135, elevation 0, left ear, lambda=0.65

Original 256-tap FIR

48

36

24

12


116/189

20 30 40 50 60 70 80 90 100 110 1200

2

4

6

8

10

12

14

16

18

20

Number of Filter Coefficients

SpectralDistanceMeasure

FIR

IIR

WIIR


117/189

1 2 3

10

20

30

40

50

60

70

80

90

Listening test results: azimuth angles 0, 135

FilterOrder

HRTF Approximation Type

FIR IIR WIIR


118/189


119/189


120/189

103

104

30

25

20

15

10

5

0

5

10

15

20

Frequency / Hz

RelativeMagnitude/dB

Modeling of minimumphase HRTFs: azi=40deg, person: 3

Minimumphase OriginalFIR, order: 48

IIR, design: Prony, order: 24WIIR, lambda= 0.65, design: Prony, order: 24


121/189

70

80

90

3.1908

FIR, azi=40, id=3

70

80

90

5.1938

70

80

90

5.4452

LL(L)/phons

7080

90

7.1978

0.2 1 3 10 21

70

80

90

9.0677

3.0824

IIR, azi=40, id=3

5.8484

7.1516

8.6572

0.2 1 3 10 21

9.7706

Frequency / kHz

0.96047

WIIR, azi=40, id=3

97

1.9814

65

2.433

49

4.4735

33

0.2 1 3 10 21

7.4997

17


122/189

70

80

90

3.1241

FIR, azi=40, id=3

70

80

90

5.0635

70

80

90

5.1856

LL(R)/phons

7080

90

6.3121

0.2 1 3 10 21

70

80

90

8.1712

2.4967

IIR, azi=40, id=3

5.293

6.2493

7.4215

0.2 1 3 10 21

8.2394

Frequency / kHz

0.7613

WIIR, azi=40, id=3

97

1.7185

65

2.1406

49

5.0108

33

0.2 1 3 10 21

7.3156

17


123/189

1

0.5

0FIR, azi=40, id=3

1

0.5

0

1

0.5

0

ITD/ms

1

0.5

0

0.2 0.5 11

0.5

0

IIR, azi=40, id=3

0.2 0.5 1Frequency / kHz

WIIR, azi=40, id=3

97

65

49

33

0.2 0.5 1

17


124/189

10 20 30 40 50 60 70 80 90 1002

3

4

5

6

7

8

Number of filter coefficients

Compositemodelingerror

Subjects: 9, azims: 4

FIRIIRWIIR


125/189


126/189


127/189


128/189

PERSON

9876543219

5%C

IPredictedValueforLOCALIZATION

5.0

4.5

4.0

3.5

3.0

2.5

2.0

1.5

1.0

TYPE

FIR

IIR

WIIR

PERSON

98765432195%C

IPredictedValueforTIMBRE

5.0

4.5

4.0

3.5

3.0

2.5

2.0

1.5

1.0

TYPE

FIR

IIR

WIIR


129/189

FILTSIZ

25797654933179

5%C

IPredictedValueforLOCALIZATION

5.0

4.5

4.0

3.5

3.0

2.5

2.0

1.5

1.0

TYPE

FIR

IIR

WIIR

FILTSIZ

257976549331795%C

IPredictedValueforTIMBRE

5.0

4.5

4.0

3.5

3.0

2.5

2.0

1.5

1.0

TYPE

FIR

IIR

WIIR


130/189


131/189


132/189

102

103

104

80

60

40

20

0

20

40

60

21.8778

21.5842

21.6527

21.3595

21.2569

19.696

Frequency (Hz)


Model: Yulewalk, order: 8, person: 1, azimuth: 20, elevation: 0, right ear

256tap orig

Bark

ERB

1/3 oct

1/10 oct

UnsmoothedWindowed FIR


133/189

102

103

104

80

60

40

20

0

20

40

60

14.2365

16.3287

16.3926

16.6194

17.9536

19.696

Frequency (Hz)


Model: BMT, order: 8, person: 1, azimuth: 20, elevation: 0, right ear

256tap orig

Bark

ERB

1/3 oct

1/10 oct

UnsmoothedWindowed FIR


134/189


135/189


136/189


137/189


138/189


139/189


140/189


141/189


142/189


143/189

Measured HRTF, subject: jyhuMinimum-phase reconstruction

Gardner approximationCooper approximation

Measured HRTF, subject: jyhuMinimum-phase reconstructionGardner approximationCooper approximation

103

104

-20

0

20Magnitude responses of 30cross-talk canceling filters 1/(Hi(z)+Hc(z))

Frequency (Hz)

Magnitude(dB)

103

104

-20

0

20Magnitude responses of 30cross-talk canceling filters 1/(Hi(z)-Hc(z))

Frequency (Hz)

Magnitude(dB)


144/189

Measured HRTF, subject: jyhuMinimum-phase reconstructionGardner approximationCooper approximation

Measured HRTF, subject: jyhu

Minimum-phase reconstructionGardner approximationCooper approximation

103

104

0

1

2

3

Frequency (Hz)

Groupdelay(m

s)

Group delay of 30cross-talk canceling filters 1/(Hi(z)+Hc(z))

103

104

0

1

2

3

Frequency (Hz)

Groupdelay(ms)

Group delay of 30cross-talk canceling filters 1/(Hi(z)-Hc(z))


145/189


146/189


147/189

xl

xr

B

A

xl

xr


148/189


149/189


150/189


151/189


152/189


153/189

xl

xr

xl xr-d =+

-

-

+

Hl

Hr

l = dxl ,p

r = dxr,p

xr,p

xl ,p

+

-


154/189

xr,p

xl ,p

+

-

-+

delay

delay

Leftplacement

filter

Rightplacement

filter

xl 0

xr0


155/189


156/189


157/189


158/189


159/189


160/189

All subjects

Magnitude mean

All subjectsMagnitude mean

102

103

104

-20

0

20Crosstalk canceling design for +-10, 1/(Hi(z)+Hc(z)), 33 subjects

Amplitude(dB

)

102

103

104

-20

0

20Crosstalk canceling design for +-10, 1/(Hi(z)-Hc(z)), 33 subjects

Frequency (Hz)

Amplitude(dB)


161/189

Test subject: jyhuCooper approximation

Gardner approximationGeneralized min-phase VL

Test subject: jyhuCooper approximationGardner approximationGeneralized min-phase VL

103

104

-20

0

20VL design 10-90, left filter, shuffler structure

Magnitude(d

B)

103

104

-20

0

20VL design 10-90, right filter, shuffler structure

Frequency (Hz)

Magnitude(dB)


162/189


163/189


164/189


165/189


166/189


167/189


168/189


169/189


170/189


171/189


172/189


173/189


174/189


175/189


176/189


177/189


178/189


179/189


180/189


181/189


182/189


183/189


184/189


185/189


186/189


187/189


188/189


189/189

virtual acoustics and 3d sound in multimedia signal processing

Documents