4 3 kettler end to end

End-to-End Speech Quality Testingin a Complex Transmission Scenario

F. Kettler*, H.W. Gierlich*,J. Berger**, H. Klaus**, I. Kliche**, K.-D. Michael**,

T. Scheerbarth**, R. Scholl***, J.-L. Freisse****

* HEAD acoustics GmbH, Herzogenrath,** T-Nova Dt. Telekom Innovationsgesellschaft mbH Berkom,

*** European Telecommu. Standards Institute ETSI (now Siemens AG)**** European Telecommunications Standards Institute ETSI

Overview

n Introduction 1st ETSI Speech Quality Test Event, Oct. 2000

n Test Setup for End-to-end Speech Quality Testing

n Test Methods, Parameters and Results

n One-way Speech Quality

n Echo and Double Talk Performance

n Background Noise Transmission Quality

n Summary

Introduction

IP networkdesigned for ‘pure’ data

transmission

• delay is high, unpredictable and time-variant• influence on echo and conversational quality• speech coders• specific kind of disturbances (packet loss, jitter...)

+ differentiation between terminal and network is nolonger possible!

Telephone networkreal time transmission

properties

ETSI Speech Quality Test Event

Organized by ETSI, Sophia Antipolis, France, Oct. 2000Sponsored by Texas Instruments, Alcatel, T-Nova and HEAD acousticsTest labs T-Nova and HEAD acoustics

n Evaluation of end-to-end (mouth to ear) speech qualityusing real equipment in a controlled VoIP environment

n One exclusive testing day for each manufactures

n Objective tests and subjective test (offline)

n Detailed test report and an anonymous reportcontaining all results

n “Goals”: identification of system improvements andcurrent status of achievable VoIP quality

Overview

n Introduction 1st ETSI Speech Quality Test Event, Oct. 2000

n Test Setup for End-to-end Speech Quality Testing

n Test Methods, Parameters and Results

n One-way Speech Quality

n Echo and Double Talk Performance

n Background Noise Transmission Quality

n Summary

Typical Signal Processing

Decoder

Buffer,Packe-tization

Buffer,PLC

H(f )

- CoderNLPNR

+

CN

VAD

EC

IP

IP

loudspeaker

jitter buffers,PLC

decodercomfortnoise

microphone

EC

levelswitching

coderVAD

noisereduction

delay,jitter,packetloss

Speech Quality Parameters

speechquality

talkingsituation

listeningsituation

conversationalsituation

… from the user’s perspective

Listening Situation

• voice activity detection• speech coders• packet loss• packet loss concealment• ...

IP

terminal A terminal B

Talking Situation

• round trip delay• echo level and echo characteristic• implementation of speech echo cancellers• quality of background noise transmission

IP

terminal A terminal B

...echo?

Conversational Situation

• propagation delay - dynamics impairments• double talk detection (EC implementation)• echo during double talk• level variations during double talk

IP

terminal A terminal B

Test Set-up

„Acoustical –acoustical“test set-up

with IP terminalsmounted to theHATS (Head AndTorso Simulators)

NISTNet

PacketMONITOR

Packet LossDELAY

Output

HEAD acousticsTestsystem CAS

Input

IP IP

IP-Phone

PCIP-Terminal

Test Set-up

„Acoustical –electrical”test set-up

with one IP phoneand one gateway(accessed througha 4-wire ISDN line)

IP

NISTNet

PacketMONITOR

T1 / E1 / BRI

Packet LossDELAY

Output

HEAD acousticsTestsystem CAS

Input

GatewayIP

ISDN DSS1

ALCATEL PABX 4400

IP

IP Terminal(Phone or PC)

Overview

n Introduction 1st ETSI Speech Quality Test Event, Oct. 2000

n Test Setup for End-to-end Speech Quality Testing

n Test Methods, Parameters and Results

n One-way Speech Quality

n Echo and Double Talk Performance

n Background Noise Transmission Quality

n Summary

Listening Situation

Analysis methods: perceptual speech quality models

IP

terminal A terminal B

feeding recording and analysis

Instrumental Measures

Realizations

� PESQ (P.862) Perceptual Evaluation of Speech Quality� TOSQA Telecommunications Objective Speech

Quality Assessment� TOSQA 2001 Extension to acoustical measurements

Typical Processing Steps (Schematic):

Adaptation Hearing Model

Comparison,Reference,Reduction,Signal Value

Results ofListening Tests!

S indexQ

SpeechSignal

Test Object Hearing Modelprocessed

signal

reference signal

1st ETSI SQTE - Test Conditions

IP

terminal A terminal B

c1: clear conditionc2: 1%packet lossc3: 2%packet lossc4: 3%Packet lossc5: 5%packet lossc6: 1%packet loss + 20 ms jitter

1st ETSI SQTE - Results

C1 C3 C5C2 C4 C6 G.711 G.729

MNRU

18 30C1 C3 C5C2 C4 C6 C1 C3 C5C2 C4 C6

G.723.1@6.3

241206

direkt

References

1

2

3

4

5

G.711

PL=20...30ms

VAD on, PLC on/off

G.723.1@6.3

PL=30ms

VAD on, PLCon/off

G.729

PL=10...30ms

VAD on, PLC on/off

G.711 reference

G.723.1@6.3 ref.

G.729 ref.

TMOS

Analysis of packet loss and PLC

Occurrence of packet loss(20 ms packet length)

Packet loss concealment(typical implementation)

Detailed analysis of PLC implementations

Cross correlation analysisvs. time

Relative Approach analysisvs. time and frequency

Overview

n Introduction 1st ETSI Speech Quality Test Event, Oct. 2000

n Test Setup for End-to-end Speech Quality Testing

n Test Methods, Parameters and Results

n One-way Speech Quality

n Echo and Double Talk Performance

n Background Noise Transmission Quality

n Summary

Analysis methods: echo measurements based onComposite Source Signals (CSS) under single anddouble talk conditions

IP

terminal A terminal B

...echo?

feeding, recording

and analysis

Talking Situation

Noise part

Voiced part

Pause

• Signal descriptionin ITU-T P.501

• Used e.g. in G.168(Echo Cancellers)

Composite Source Signal

Sequence length 700 ms

SLR JLR JLR RLR7dB 0dB 0dB 3dB

TELR a(Echo) a(Echo) TELR

RLR JLR JLR SLR3dB 0dB 0dB 7dB

OLR = 10dB

Requirement on echo lossdepends on transmission

delay

Echo under Single Talk Conditions

Analysis methods: double talk measurements basedon two de-correlated Composite Source Signals

IP

terminal A terminal B

feeding feeding, recording

and analysis

Conversational Situation

• Signal description in ITU-T P.501• Analysis methods in ITU-T P.502

Combination of two CSS

Typical double talksequence (speech)English, male and femalevoices

Combination of CSSincl. single and doubletalk periods

Comparison to Real Speech

measured signalnearly perfectlymatches originaltest signal

green: measured signal

red: original test signal

yellow: overlap

Double Talk Evaluation: Result #1

green: measured signal

red: original test signal

yellow: overlap

Double Talk Evaluation: Result #2

short term echocomponentsduring pausesand jitter

echo jitter

green: measured signal

red: original test signal

yellow: overlap

Double Talk Evaluation: Result #3

strong echocomponents(occurred onlyduring doubletalk!)

strong echo

green: measured signal

red: original test signal

yellow: overlap

Double Talk Evaluation: Result #4

clipping atbeginning andend of mostbursts andcomfort noiseduring pauses

clipping

comfort noise

Overview

n Introduction 1st ETSI Speech Quality Test Event, Oct. 2000

n Test Setup for End-to-end Speech Quality Testing

n Test Methods, Parameters and Results

n One-way Speech Quality

n Echo and Double Talk Performance

n Background Noise Transmission Quality

n Summary

Test Signal for Noise Transmission

Noise signal(Hoth spectrum)with increasinglevel vs. time

Signal descriptionin ITU-T P.800

Comparison of Test Results

1

2

3

4

Comparison ofthree typicalimplementations

(1) red: original test signal

(2) yellow: VAD threshold

(3) blue: AGC gain

(4) magenta: comfortnoise injection

Overview

n Introduction 1st ETSI Speech Quality Test Event, Oct. 2000

n Test Setup for End-to-end Speech Quality Testing

n Test Methods, Parameters and Results

n One-way Speech Quality

n Echo and Double Talk Performance

n Background Noise Transmission Quality

n Summary

n Identification of improvements and current status ofachievable VoIP quality - “success”

n Combination of various test procedures!

n Overall quality rating given by the participants“excellent”

n A short report can be found under etsi.org/plugtests

n ETSI strongly recommended to continue the processof end-to-end speech quality tests

n 2nd SQTE planned for April 2002

Summary